ABSTRACT
Title of dissertation: COCHLEAR IMPLANTATION:
PATH PLANNING ALGORITHMS
AND DYNAMICS
Celeste R.C. Poley
Doctor of Philosophy, 2019
Dissertation directed by: Professor Balakumar Balachandran
Assistant Professor Axel Krieger
Department of Mechanical Engineering
The focus of this dissertation is on incorporating robotics into pediatric cochlear
implantation surgery. Since the 1980s, over 300,000 cochlear implantation surgeries
have been performed worldwide, both in adults and children alike. For this disserta-
tion research, surgical constraints in the operating theater are of utmost importance
for the health and safety of the patient. As the field moves toward minimally inva-
sive surgery, the issues that come with this, such as the loss of the natural field of
view and the loss of tactile sense can create significant hurdles for surgeons. Medical
robotics can be used to decrease the limitations of such surgical procedures since a
desirable attribute of surgical robots is dexterity. Medical robotics can be used to
can be used to counter these limitations, by taking advantage of the dexterity of sur-
gical robots. These robots can be used in complex working environments for surgical
procedures such as cochlear implantation surgery (CIS). The author?s dissertation
contains simulation, analytical, and numerical research, through which the effects of
dynamics within the path planning algorithms on simulated and modeled cochlear
implantation surgery have been studied. A novel path planning algorithm has been
developed by making use of Rapidly-exploring Randomized Trees (RRT), and sub-
sequently incorporating Sequential Quadratic Programming. The goal in utilizing
a path planning algorithm (PPA) would be to increase safety and aid surgeons in
a tightly constrained environment of pediatric temporal bone, which differs in ge-
ometry and size from adult temporal bones, and to positively impact the surgical
procedure and recuperation from surgery. This algorithm was chosen for use in the
tight spaces presented by pediatric patient anatomy and to address patient specific
constraints or abnormalities that arise with cochlear implantation surgery in cases
of congenital deafness, to add an extra layer of safety for patients. This method
allows for more torque handling in tiny and heavily constrained environments, and
prevents nicking of delicate anatomy, such as the cranial facial nerve, which can
cause facial muscle paralysis if exposed to the slightest damage. The testing of the
planning algorithm is carried out in a programming environment. These models
are based on geometric equations describing the inner ear anatomy, based on data
collected as a part of this dissertation work. Through this doctoral dissertation re-
search, the author has developed several novel methods and innovative techniques to
improve associated path planning algorithm. Since cochlear implantation surgeries
are moving in the direction of minimally invasive surgery, it would be a beneficial
goal to improve the surgery by including a path planning algorithm and a simulated
robotic system to help reach the dissertation?s goal of simulating the drilling done
for cochlear implantation surgery, and with the ultimate goal of improving patient
outcome and minimizing time to recovery.
COCHLEAR IMPLANTATION: PATH PLANNING
ALGORITHMS AND DYNAMICS
by
Celeste R.C. Poley
Dissertation submitted to the Faculty of the Graduate School of the
University of Maryland, College Park in partial fulfillment
of the requirements for the degree of
Doctor of Philosophy
2019
Advisory Committee:
Professor Balakumar Balachandran, Mechanical Engineering, Chair & Advisor
Assistant Professor Axel Krieger, Mechanical Engineering, Co-Advisor
Professor Olivier Bauchau, Aerospace Engineering (Dean?s Representative)
Professor Amr Baz, Mechanical Engineering
Associate Professor Nikhil Chopra, Mechanical Engineering
Assistant Professor Yancy Diaz-Mercado, Mechanical Engineering
? Copyright by
Celeste R.C. Poley
2019

Acknowledgments
I would like to thank Professor Balakumar Balachandran for giving me an
invaluable opportunity to work on challenging and extremely interesting projects
over the past several years. He has always made himself available for help, and
advice in dealing with a complex problem. It has been a pleasure to work with
and learn from such an extraordinary individual. I would also like to thank my
co-advisor, Dr. Axel Krieger for his invaluable medical robotics expertise and great
and insightful advice for many aspects of this project.
Thanks are due to Professor Bauchau, Professor Chopra, Professor Diaz-
Mercado, and Professor Baz, for agreeing to serve on my dissertation committee
and for reviewing this dissertation. I would also like to thank Professor Preciado
of the George Washington University for allowing me to better understand the sur-
gical procedure through shadowed surgeries, explanations of inner and middle ear
anatomy on CT scans, and explanations regarding the nuances of the procedure
itself.
I owe my deepest thanks to my parents for all of their generous support over the
years. Many thanks are also given to my lab-mates and coworkers at my internship at
Sheikh Zayed Institute of Pediatric Surgical Innovation for their thoughtful advice,
and for their wonderful friendship. It is impossible to remember all, and I apologize
to those I?ve inadvertently left out.
Support received for this research through the NSF Graduate Fellowship Re-
search Program Grant DGE 1322106, NSF INTERN Supplemental Funding Grant
ii
DGE 1322106, the LSAMP Bridge to the Doctorate Program Grant, the Sloan
Foundation NACME Award, and the Dean?s Fellowship from the Department, and
is gratefully acknowledged. The pandemic delayed submission of the final form of
this dissertation.
Lastly, thank God!
iii
Table of Contents
Preface ii
Foreword ii
Dedication ii
Acknowledgements ii
Table of Contents iv
List of Tables vii
List of Figures viii
List of Abbreviations xi
1 Introduction 1
1.1 Problem of Interest . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1.0.1 Organization of this Chapter . . . . . . . . . . . . . 4
1.2 Prior Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.2.1 Robotic Motion Planning . . . . . . . . . . . . . . . . . . . . 5
1.3 Models and Governing Equations . . . . . . . . . . . . . . . . . . . . 8
1.3.1 System Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . 8
1.3.2 Robotic Cochlear Implantation . . . . . . . . . . . . . . . . . 9
1.4 Planning Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1 Robot 3D Path Planning Algorithms . . . . . . . . . . . . . . 13
1.4.2 Path Planning Algorithms and Cochlear Implantation . . . . . 14
1.4.3 Summary of Previous Work . . . . . . . . . . . . . . . . . . . 16
1.5 Clinical Information . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.5.1 Types of Cochlear Implantation Surgery . . . . . . . . . . . . 19
1.5.2 Steps to General Surgical Procedure . . . . . . . . . . . . . . 21
1.6 Description of Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 22
1.7 Outline of Dissertation . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2 RRT Kinematics, Dynamics, and Obstacle Avoidance 25
2.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2 Experimental and Computational Background . . . . . . . . . . . . . 26
iv
2.2.1 Accounting for Constraints . . . . . . . . . . . . . . . . . . . . 26
2.2.2 Initial States and Goal states for System . . . . . . . . . . . . 34
2.2.3 Generation of Path Planning Algorithms . . . . . . . . . . . . 35
2.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
2.3.1 Two Link Manipulator Arm (4 DOF) Case . . . . . . . . . . . 38
2.3.1.1 Kinematics . . . . . . . . . . . . . . . . . . . . . . . 39
2.3.1.2 Dynamics . . . . . . . . . . . . . . . . . . . . . . . . 42
2.3.2 LWR Case . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
2.3.2.1 Kinematics for KUKA LWR IV+ (LWR) . . . . . . . 45
2.3.3 Rapidly-exploring Random Trees . . . . . . . . . . . . . . . . 48
2.4 Cochlea Geometric Constraints as Obstacles . . . . . . . . . . . . . . 57
2.4.1 Two Links, Four DOF Case . . . . . . . . . . . . . . . . . . . 58
2.5 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
3 Image Processing and Segmentation and Observation of Surgical Procedures
for Implementation in PPA 71
3.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.2.1 C.H.A.R.G.E. Patients . . . . . . . . . . . . . . . . . . . . . . 74
3.3 Importance of Anatomical Accuracy . . . . . . . . . . . . . . . . . . . 74
3.4 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
3.4.1 Segmentation of Inner Ear . . . . . . . . . . . . . . . . . . . . 76
3.4.1.1 Proposed Patient Specific Drilling (Cutting) Paths . 82
3.4.2 Methodology for Generating Surface Fitting Equations in MAT-
LAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
3.4.3 Body-Centric Frame to World Frame . . . . . . . . . . . . . . 88
3.4.4 Dimensions of cochlea . . . . . . . . . . . . . . . . . . . . . . 91
3.4.5 Temporal Bone CT scan generation . . . . . . . . . . . . . . . 93
3.5 Surface Fitting Equations to Map Out Cochlea . . . . . . . . . . . . 95
3.5.1 Semicircular Canals . . . . . . . . . . . . . . . . . . . . . . . . 95
3.5.2 Cochlea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3.5.3 Full cochlea model . . . . . . . . . . . . . . . . . . . . . . . . 98
3.6 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4 Cutting Path Planning Algorithms and Updating with Anatomically Correct
Obstacles 102
4.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
4.1.1 Governing Equations for Sequential Quadratic Programming
(SQP) Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . 105
4.1.1.1 Robot Trajectory Governing Equations . . . . . . . . 105
4.1.1.2 Cutting Path Governing Equations . . . . . . . . . . 107
4.2 Task Space Dynamics . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.3 Sequential Quadratic Programming vs. Rapidly-exploring Random-
ized Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
4.3.1 Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110
v
4.3.2 Constraints for the Cutting Path . . . . . . . . . . . . . . . . 111
4.3.3 Constraints for the tool path . . . . . . . . . . . . . . . . . . . 113
4.3.4 Pseudocode For Sequential Quadratic Programming . . . . . . 118
4.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
4.4.1 Results for Robot Trajectory . . . . . . . . . . . . . . . . . . . 118
4.5 Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
5 Concluding Remarks 123
5.1 Summary of Contributions . . . . . . . . . . . . . . . . . . . . . . . . 123
5.2 Recommendations for Future Work . . . . . . . . . . . . . . . . . . . 124
5.3 Additional Remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
A Medical Terminology and Images 127
B Additional Data 136
B.1 Figures showing measured dimensions . . . . . . . . . . . . . . . . . . 136
B.2 MATLAB code for generating surface equations . . . . . . . . . . . . 136
Bibliography 150
vi
List of Tables
2.1 Initial and Goal States for Two Link, 4 DOF System (Before con-
straints are taken into effect). . . . . . . . . . . . . . . . . . . . . . . 34
2.2 Initial and Goal States for 7 Link, 7 DOF System (Before constraints
are taken into effect). . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
2.3 Initial and Goal States for 7 Link, 7 DOF System (Before Constraints
Taken into Effect). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
2.4 RRT PLANNING ALGORITHM PSEUDOCODE. . . . . . . . . . . 54
2.5 RRT PLANNING ALGORITHM WITH OBSTACLE AVOIDANCE. 55
2.6 Logarithmic spiral equation with arbitrary constant values. . . . . . . 58
2.7 Average dimensions of cochlea in adult humans. . . . . . . . . . . . . 58
3.1 Dimensions for Boundary Conditions on cochlea from patient data sets 92
3.2 Polynomial equation coefficients for semicircular canals. . . . . . . . . 95
3.3 Statistics on Surface Fitting Equations for Semicircular Canals. . . . 96
3.4 Polynomial equation coefficients for cochlea. . . . . . . . . . . . . . . 98
3.5 Statistics on Surface Fitting Eqns for cochlea. . . . . . . . . . . . . . 98
3.6 Polynomial equation coefficients for full cochlea. . . . . . . . . . . . . 99
3.7 Statistics of Surface Fitting equations for full cochlea. . . . . . . . . . 100
4.1 Dimensions used to generate boundary conditions for the cutting path
(Sample 3 Data). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
vii
List of Figures
1.1 Ear anatomy, adapted from Hawkins (2019). . . . . . . . . . . . . . . 19
1.2 Cochlear implant demonstrating different parts, adapted from NIH
(2018). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.1 Desired robot trajectory for each link. . . . . . . . . . . . . . . . . . . 39
2.2 Kinematics case: Angular positions and angular speeds for ?1, ??1, ?2,
??2, ?1, ??1, ?2, and ??2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
2.3 Kinematics case: Torques associated with states ?1, ?2, ?1, and ?2. . . 41
2.4 Dynamic case: Angular positions and angular speeds for ?1, ??1 , ?2 ,
??2, ?1, ??1, ?2, and ??2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
2.5 Dynamic case: Torques associated with states ?1, ?2, ?1, and ?2. . . . 44
2.6 KUKA DLR LWR IV+ Reference Diagram for Robot Trajectories:
Angles ?1, ?2, ?3, ?4, ?5, ?6, and ?7 are shown in proper orientation. . 46
2.7 Kinematics case: Angular positions for LWR associated with ?1, ?2,
?3, ?4, ?5, ?6, and ?7. . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.8 Kinematics case: Angular speeds for LWR associated with ??1, ??2, ??3,
??4, ??5, ??6 and ??7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.9 Kinematics case: Torques for LWR associated with states ?1, ?2, ?3,
?4, ?5, ?6, and ?7. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
2.10 RRT Advantages and Disadvantages. . . . . . . . . . . . . . . . . . . 52
2.11 RRT example. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
2.12 Simulated cochlea for RRT studies with obstacle avoidance. . . . . . . 59
2.13 Projection of state space plot in ?1-??1 Space. . . . . . . . . . . . . . . 60
2.14 Projection of state space plot in ?2 - ??2 space. . . . . . . . . . . . . . 61
2.15 Projection of state space plot in ?1-??1 space. . . . . . . . . . . . . . . 62
2.16 Projection of state space plot in ?2-??2 space. . . . . . . . . . . . . . . 63
2.17 Angular position histories For ?1, ?2, ?1 and ?2. . . . . . . . . . . . . 66
2.18 Angular speed histories for ??1 and ??2. . . . . . . . . . . . . . . . . . . 67
2.19 Angular speed histories for ??1 and ??2. . . . . . . . . . . . . . . . . . . 68
3.1 General flow for 3D segmentation process. . . . . . . . . . . . . . . . 76
3.2 Axial view (red slice) of sample No. 3 of CT scan of temporal bone
on the left side of a patient. . . . . . . . . . . . . . . . . . . . . . . . 77
viii
3.3 Coronal view (green slice) of sample No. 3 of CT scan of temporal
bone on the left side of a patient. . . . . . . . . . . . . . . . . . . . . 77
3.4 Sagittal view (yellow slice) of sample No.3 of CT scan of temporal
bone on the left side of a patient. . . . . . . . . . . . . . . . . . . . . 78
3.5 Representation of Slicer environment. . . . . . . . . . . . . . . . . . . 78
3.6 Segmented model of sample No 1. cochlea and round window, bony
labyrinth showing abnormal anatomical formations. . . . . . . . . . . 79
3.7 Landmarks and anatomy of a normal cochlea Gray, (2022). . . . . . . 81
3.8 View of the cranial facial nerve and the round window generated from
sample No. 3. The different viewpoints of the inner ear models to
show the complexity of the delicate anatomy and the tight environ-
ment faced by surgeons. . . . . . . . . . . . . . . . . . . . . . . . . . 81
3.9 Slice from CT scan for patient 1, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed 2D repre-
sentation of the cutting path. . . . . . . . . . . . . . . . . . . . . . . 83
3.10 Slice from CT scan for patient 2, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed
2D representation cutting path. . . . . . . . . . . . . . . . . . . . . . 84
3.11 Slice from CT scan for patient 3, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed
2D representation cutting path. . . . . . . . . . . . . . . . . . . . . . 85
3.12 Slice from CT scan for patient 4, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed
2D representation cutting path. . . . . . . . . . . . . . . . . . . . . . 86
3.13 Slice from CT scan for patient 5, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed
2D representation cutting path. . . . . . . . . . . . . . . . . . . . . . 86
3.14 General flow for surface fitting. . . . . . . . . . . . . . . . . . . . . . 87
3.15 RAS Coordinate System is a body-centric coordinate frame utilized
by CT imaging systems. . . . . . . . . . . . . . . . . . . . . . . . . . 88
3.16 Cochlea and semicircular canals shape validated in point cloud plot
in MATLAB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
3.17 Temporal bone of sample No. 3 showing the orientation of the anatomy
during the surgical procedure, along with a marker showing the loca-
tion for the initial state Qi located on the temporal bone. . . . . . . 93
3.18 Model generated of sample No. 3, showing the orientation of the
anatomy during the surgical procedure, with a marker showing the
location for the goal state (Qg = round window). . . . . . . . . . . . 94
3.19 Temporal bone of sample No. 3. . . . . . . . . . . . . . . . . . . . . . 94
3.20 Curve fitting generated in MATLAB based on point cloud data of
semicircular canals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
3.21 Polynomial equation coefficients for the cochlea. . . . . . . . . . . . . 97
3.22 Curve fitting generated in MATLAB based on point cloud of the full
cochlea for sample No. 3. . . . . . . . . . . . . . . . . . . . . . . . . . 99
ix
4.1 Posterior view mastoid process to see the inner ear, adapted from
CE4RT, (2019). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
4.2 Generalized drilling path, axial view Scorza et al., (2021). . . . . . . . 105
4.3 Desired drilling path to be generated by SQP. . . . . . . . . . . . . . 106
4.4 Angular position (? i) of the KUKA LWR in the SQP optimized system.119
4.5 Angular speed ?? i of the KUKA LWR in the SQP optimized system. 120
4.6 Minimized torque in the SQP optimized system. . . . . . . . . . . . . 121
A.1 External Auditory Canal, adapted from Trust (2019.) . . . . . . . . . 127
A.2 Reference Planes and Frames, adapted from College (2019). . . . . . 128
A.3 Body-centric reference frames- coronal, sagittal, and frontal, adapted
from Terray (2016). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
A.4 Coronal Section View of the Mastoid Air Cells, adapted from Gray
(2019). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
A.5 Interior Portions of the cochlea, adapted from Park (2019). . . . . . . 130
B.1 The polynomial fit for the segmented models of the cochlea generated
in MATLAB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
B.2 Segmented Model of Sample No. 2. cochlea and round window. . . . 137
B.3 Segmented Model of Case No 4. Cochlea and Round Window, Semi-
circular Canals show more mild malformations than in Case No 1. . . 138
B.4 Sample No. 1 cochlea and round window segmentation. . . . . . . . . 139
B.5 Sample No. 2 cochlea and round window segmentation. . . . . . . . . 139
B.6 Sample No. 4 cochlea and round window segmentation. . . . . . . . . 140
B.7 Sample No. 5 cochlea and round window segmentation. . . . . . . . . 141
B.8 The polynomial fit for the segmented models of the semicircular
canals generated in MATLAB. . . . . . . . . . . . . . . . . . . . . . . 141
B.9 Length of cranial facial nerve (CN7). . . . . . . . . . . . . . . . . . . 142
B.10 Approximate diameter of cranial facial nerve (CN7). . . . . . . . . . . 142
B.11 Diameter of the round window membrane (RW). . . . . . . . . . . . . 143
B.12 Approximate length of the temporal bone from Sample No. 3. . . . . 143
B.13 Width of cochlea. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
B.14 Length of cochlea. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
x
List of Abbreviations
Path Planning Terms
Path Planning Terms
Ai Robot link i
C Work space Region, belonging to W
Cfree Free space belonging to the Work space
Cobs Obstacle region belonging to C
RRT Rapid -exploring Randomized Trees
O Obstacle region belonging to W
PPA Path planning algorithms
Qi Initial state
Qg Goal state
Qnear subset of Nodes closest to Qg
Qnew New node(s) to be added to RRT tree
Qrand unconnected node, randomly chosen
W World space, includes work space C
Dynamics and Robotics Terms
B Control input matrix, n?m
D Mass and Inertia Matrix, n? n
T , set of all torques, n? 1
? , singular torque contained in T , m? n
DOF Degree(s) of freedom
EOM Equations of motion
Jh Constraint Jacobian, p?m
IIWA KUKA IIWA
I local Inertia
LWR KUKA LWR IV+
qi Generalized coordinates
T Torque vector, n? 1
T o Optimal torque vector, n? 1
? constraint term, p? 1
SQP Sequential Quadratic Programming
Clinical Terms
BT Basal Turn of the Cochlea
CI Cochlear implantation
CIS Cochlear implantation surgery
CN7 Cranial Nerve 7, Cranial Facial Nerve
CT Computed tomography
IGS Image Guided software
MIS Minimally invasive surgery
RCIS Robotic cochlear implantation surgery
RWE Round window enlargement
RW Round window
TB Temporal Bone
xi
xii
Chapter 1: Introduction
1.1 Problem of Interest
In this dissertation, the author has studied path planning algorithms that
have been developed towards the eventual goal of improving cochlear implantation
surgery. Since cochlear implantation surgery was first performed in the 1980s, and
over 7,000 cochlear implantation surgeries are performed annually in the United
States, see Megerian, (2019). These surgeries are generally performed on patients
who are congenitally deaf, or have suffered profound hearing loss, see, for example,
Kliegman, et al. (2007). A number of these surgical cases are pediatric cases, and
it is important to note that the geometry of the inner ear of pediatric patients is
geometrically different from that of adults (McRackan et al., 2012). It has been
shown that early surgical intervention for an infant candidate to receive a cochlear
implant can greatly improve their speech development and dramatically improve
their academic, social and economic successes later in life, see, for example, Carlson
(2018) and Kliegman et al., (2007).
It is important to note that the geometry of the inner ear of pediatric patients
is geometrically different from that of adults. The authors in McRackan et al., (2012)
have noted in their research that surgeons often feel that CIS is more complicated
1
in pediatric cases due to the changes in physiology of the inner and middle ear.
Therefore, it is of interest to study pediatric cases.
A cochlear implant surgery is performed in a part of the body that is delicate
and full of complex and critically sensitive anatomy, see for example, (Kronenberg
et al., 2001). While a skilled surgeon is accomplished at performing this surgery,
there is a learning curve for carrying out this surgery, and the slightest insult to a
number of nerves, such as the cranial facial nerve, can lead to facial muscle paralysis
or some other devastating conditions; see Kronenberg et al., (2001); Labadie et al.,
(2014).
The motivation for this dissertation work is to develop and to test a path plan-
ning algorithm that would aid in robotic cochlear implantation surgery. There were
a variety of components that were needed even for the simulation of this surgery.
In early renditions, both the tool and the cutting path planning algorithms were
merged. After careful consideration, the path planning algorithm chosen for utiliza-
tion in generating the path was a probabilistic based algorithm called Randomly -
exploring Randomized Trees (RRT). This was chosen for several reasons. The first
of which is the fact that this algorithm has a high adaptable for use with governing
equations of motion for systems with a high number of degrees of freedom (DOF)
found in LaValle, (2014). Another reason is the fact that this algorithm was adapt-
able for the inclusion of obstacle avoidance and collision detection with the given
differential constraints. Additionally, obstacle avoidance equations were used in the
algorithm so that anatomically accurate models of the inner ear models could be
incorporated into the path planning algorithms. This approach provided for the
2
anatomically correct models that were generated from Temporal Bone CT scans
from of five anonymous pediatric cases. These models were verified across several
image processing programs (3D Slicer, Mango, and MATLAB).
The early generations of the algorithm had purely holonomic constraints (or
equality based constraints, or a constraint that can be integrated). In the recent
generation of the path planning algorithm, the author has included non-holonomic
constraints (inequality based constraints), a constraint that cannot be integrated
Greenwood, (1997). These are implemented on the velocity of the end-effector in
the simulation, as well as on parts of the unique shapes of the surfaces of the
anatomy of the inner ear. Minor smoothing was achieved with a basic low pass
filter to filter out the noise in the system for early generations of the algorithm.
In later implementations of the path planning algorithm, improvements have been
made upon this model with a sequential quadratic programming algorithm.
As the algorithm was being developed, the author of this dissertation also
decided to consider a different path planning algorithm, called Sequential Quadratic
Programming (SQP); see for example Boggs and Tolle (1996). This algorithm is
grid based as opposed to RRT, and has a great capability for handling nonlinear
constraints that can be present when simulating the surgical drilling path. This
method will be discussed in more detail later on in Chapter 4 of this dissertation.
Most current research into robotic cochlear implantation surgery (RCIS) is
either imaging based and does not include path planning algorithms, or is based
on the use of a weighted optimization algorithm; for example, Eilers et al., (2009).
One of the primary goals of this research is to explore what kinds of improvements
3
can be made to robotic cochlear implantation surgery through the use of obstacle
avoidance for critical anatomy and collision detection of the robotic system, as this
system is used to assist the surgeon throughout the procedure. This is a difficult
problem to model due to the high number of degrees of freedom (DOF) for the tool
path, and the complex modeling for the cutting path. This necessitates the use of
non-holonomic conditions for controlling the speed, and angles on the robot, image
processing, geometric modeling constraints, and distance constraints in terms of
millimeters, obstacle avoidance, and setting restrictions on surrounding areas. An
example of a holonomic constraint is a constraint on the joint that would limit the
motion along one axis such that movement along that axis would be zero (e.g., z =
0). The holonomic and non-holonomic constraints are further defined in Chapter 4.
1.1.0.1 Organization of this Chapter
The problem of interest for this work is to determine how to utilize a path plan-
ning algorithm that generates an optimized path for cochlear implantation surgery
(CIS) that is unique to each patient, that acknowledges the fact that the inner ear is
located in what is considered to be a highly cluttered environment, filled with tiny,
delicate, and critical anatomy.
The principal objective of this dissertation is to develop a path planning algo-
rithm, in which one utilizes information from each patient for developing a unique
path planning algorithm that is optimal for each cochleostomy and mastoidectomy
surgery. With the stated objective, the author has worked on building the necessary
4
modeling, analysis, and tools in order to use path planning algorithms with obsta-
cle avoidance in a three-dimensional (3D) setting for cochlear implantation surgery
(CIS). In the next section, background knowledge needed to understand how to
create the simulation is briefly reviewed. The organization of this dissertation is
discussed at the end of this chapter.
1.2 Prior Work
1.2.1 Robotic Motion Planning
Motion analysis, while in and of itself is not a new topic of research, has ap-
plications in the field of medical robotics. Since the inception of modern robotics,
robotic motion planning has been studied. As the field of medical robotics has devel-
oped, medical robots are increasingly being used to assist surgeons during procedures
requiring intricate precision. A number of different path planning algorithms have
been proposed over the last decade for surgical procedures such as minimally inva-
sive, endoscopic, and numerous other procedures; see Aghakhani, et al., (2013), and
Ko and Rodriguez (2012). As minimally invasive procedures become more common,
the surgeon?s natural field of vision is inhibited, unlike in traditional open surgery.
This can cause the surgeon to become tired, and potentially lead to other complica-
tions due to lack of visibility. With these issues in mind, path planning algorithms
have been created and tested specifically for these surgical procedures, so that with
a robotic manipulator arm the movement, dexterity, and visibility for the surgical
procedure can be enhanced (e.g. Eilers et al., 2009). In such cases, path planning
5
can be used to develop optimal paths despite the motion constraints that arise due
to the dimensions of the human body. In the context of this dissertation work,
the author strived to determine the effectiveness of a unique constraint equation on
robotic manipulator arms such as the KUKA LWR (LWR) through simulations.
Generally, the LWR is used in industrial applications; however, maneuverabil-
ity, and precision of this manipulator arm is believed to be an ideal candidate for
use in medical applications because of, amongst other things, safe physical human-
robot interactions and the system?s reactive behavior. The LWR has 7 degrees of
Freedom (DOF), which is similar to that of a human arm, making it much more
maneuverable than a traditional robot in industrial applications, see for example
(Bischoff et al., 2010). Since such a robot is much more maneuverable than tra-
ditional robots, and is geared to work specifically with humans, it is believed that
this robot would be suitable for medical applications. There is a KUKA Medical
robot, which is very similar to this robot, but for the purposes of this dissertation
the author will be using the regular LWR as the author was able to obtain the
necessary measurements, dimensions, and other necessary information for this par-
ticular robot Company, (2019). Over the past several decades, research has been
undertaken on planning for minimally invasive surgery (MIS), percutaneous needle
insertion alongside other surgical procedures; see Nageotte et al., (2005); Ko and
Rodriguez (2012); and Siciliano et al., (2016).
Surgeons still face the risks and complications that arise from cochlear implan-
tation surgery, even with the Da Vinci Surgical Suite. By drilling a fraction of a
millimeter too much, a surgeon can destroy the delicate anatomy of the inner ear
6
structures and cause irreversible damage; see Wanna et al., (2014); Zhang et al.,
(2009); and (Pile et al., 2011) for further information. Therefore, in cases such as
these, the goal of path planning for robotic procedures is to minimize risks. A num-
ber of researchers have been involved in creating pre-programmed robotic systems to
automate this procedure (as well as other procedures) to mitigate the risks. With
the rise in the use of robotic surgical tools and minimally invasive surgery, path
planning generation has become increasingly used in the medical field as a means
to reduce the amount of time that procedures take, and to avoid accidental nicks
and tears to delicate structures, improve recovery time for patients, and improve the
work environment for the surgeon Eilers et al., (2009), Ko and Rodriguez (2012).
In a paper presented by Adhami and Coste-Maniere (2003), the authors detail
a plan to use an optimization algorithm to determine the best location for ports
for robotically assisted minimally invasive surgery (RMIS), in two phases. The
first phase looks at patient dependent anatomy and the endoscopic tools, as well
as the capabilities and the limitations of the robotic system being utilized. In the
second phase constructs a path is constructed for a collision free operation of the
intervention, or as close to collision free as possible. In the paper presented by
Aghakhani, et al., (2013), the authors use a novel kinematic mathematical model to
constrain the motion of a robotic manipulator arm and endoscopic camera holder
that takes into account constraints for the port of entry for the tools Aghakhani,
et al., (2013). The goal is to determine kinematic models that will still allow for
success within the constraints applied for robotic minimally invasive surgery. In
another paper, presented by Ko and Rodriguez (2012), the authors of this study
7
use a novel device for percutaneous intervention by using a programmable bevel to
control the angles between the interlocked needle segments. The authors created a
path planning algorithm, in which obstacle avoidance is used.
A robotic cochlear implantation study was presented on a novel algorithm for
the robotic insertion of periomodiolar electrode arrays (PEAs). This study focused
in part on studying the shape variability of the PEAs to determine how to obtain
the optimal shape for insertion when using robotically guided surgical tools Pile et
al., (2011). A study was conducted on ten temporal bones to test out their unique
algorithm for robot guided surgical tools to perform the cochleostomy surgery, and
they were successful in nine out of ten cases, due to concerns about proximity to
the round window in the tenth temporal bone. Of the successful nine cases that
were completed, the chorda tympani nerve was violated in one case, and the stapes
bone was violated in two cases, please refer to Majdani et al., (2009) for further
information.
1.3 Models and Governing Equations
1.3.1 System Dynamics
As can be seen in the work presented by Gaz et al., (2014) utilize the LWR
and the Fast Research Interface software (FRI) to provide end user numerical values
of the elements of the link inertia matrix and the gravity vector at the current
robot configuration. These investigators used the Denavit-Hartenberg convention
for ascertaining information on robot dynamics; however, instead of using the Euler-
8
Lagrange method for obtaining the equations of motion (EOM), they utilized the
Newton-Euler equations of motion. The researchers essentially used a curve fitting
effort, through which one takes data from the FRI interface to collect data. The
EOM obtained in reference Gaz et al., (2014) have been used as a reference for
the EOM generated for this dissertation. The dynamics treatment follows standard
treatments provided in references Asada and Slotine (1986); Greenwood (1997); and
Spong et al., (2012).
1.3.2 Robotic Cochlear Implantation
In the work presented by Majdani et al., (2009), a robotic system was con-
structed to perform cochlear implantation surgery. Their concept was to utilize a
minimally invasive approach by drilling directly into the cochlea, with the help of
image guided software (IGS). The study was conducted on ten temporal bones. Nine
of the ten cases were successful; however, the tenth was a failure due to the struc-
ture anatomy of the temporal bone, and the proximity of the goal state to round
window (Majdani et al., 2009). In addition to using IGS software, Majdani et al.,
(2009) developed a closed loop optimization program to use in combination with
the IGS for their 6 DOF robotic system. These researchers then tested the efficacy
of their system on 10 cadaveric cochleas (Majdani et al., 2009). After conducting
these experiments, nine out of ten cadaveric test cases were successfully performed
with their 6 DOF robot.
Another research group demonstrated the ability to perform robotic cochlear
9
implantation (RCIS) surgery successfully with human patients (Caversaccio et al.
2017). They used an image guided robotic surgical system solely for the drilling of
the tunnel for the electrode array during the robotic cochlear implantation surgery
(Caversaccio et al., 2017).
Before the surgery, four 5 mm surgical screws (fiducials) are strategically
placed in the mastoid for patient registration during the CT scans. The images
are then verified for accuracy and quality to help the surgeon with the mastoidec-
tomy and cochleostomy. Once that is completed, surgical planning and drill path
placement can commence. The software program presented by Caversaccio et al.,
(2017) helps generate a safe access tunnel that will be drilled by the robotic system.
The software program is used to semi-automatically segment critical anatomical
structures of the middle ear and surrounding anatomy (facial nerve, ossicles, chorda
tympani, and so on). As the path of the chorda tympani is not typically visible on
CT images, especially through the middle ear cavity, the surgeon uses the software
to create a spline-based model of the path for the chorda tympani, by first selecting
a reference landmark on the malleus (Caversaccio et al., 2017). The ideal drilling
target is the round window (RW) membrane of the cochlea. The drilling angle and
path are determined by the software program.
At the beginning of the surgical procedures, the fiducial screws are recorded by
the optical tracking system and safety checks, as well as refinement of registration
and verification ensue. The first of three phases for drilling, which is done by an
image guided robotic system, has the drill going down to the level of the facial
nerve at a rate of 0.5 mm/s, and CT scans are done to check progress and for
10
potential issues (Caversaccio et al. 2017). In the second phase, the drilling is done
in intervals slowly as the facial nerve and chorda tympani are close in proximity.
In the third phase, the drilling reaches the target of the round window on the
cochlea. Additionally, it was found that the surgeons conducting the surgeries with
the robotic system favored the minimally invasive techniques that only required
a minimal mastoidectomy, which resulted in improved outcomes for the patient
with regard to pain, recovery, reduced risk of infection, and some other potential
complications. In addition to these benefits, having a minimally invasive (keyhole)
approach to drilling in the mastoid bone, the patients reported better ability to
stabilize pressure in their inner ears. The drilling performed was shown to have an
accuracy of 0.2mm. Additionally, it is important to note that in utilizing robotic
cochlear implantation, Caversaccio et al. (2017) were able to pinpoint the robot?s
proximity to nerves within 0.1 mm: this gave surgeons greater confidence while
performing the CIS with a robotic system.
One of the primary differences between the work presented by Caversaccio et
al. (2017) and the work of the author of this dissertation, is that in this dissertation
collision detection and obstacle avoidance are included in the updated models, in
addition to the use of path planning algorithms that do not solely rely on image
guided feedback.
11
1.4 Planning Algorithms
PPA are used in a range of applications in the robotics field. These applica-
tions, which are diverse, include protein folding, aerospace applications, manufactur-
ing, and mapping out unknown environments LaValle, (2014). PPA are useful and
beneficial in a wide range of highly complex environments, and they are expected to
be beneficial to critical medical procedures. The intent of this dissertation author?s
research is to use simulations for eventual path planning in a variety of delicate surgi-
cal procedures, such as cochlear implantation, and to aid in preventing complications
such as injury to the facial nerve, leakage of spinal fluid, swelling, and others Kro-
nenberg et al., (2001). While complications from the cochlear implantation surgery
are rare, some of these might be preventable with a PPA that automatically restricts
the robot and surgical tools from approaching anatomy that has been defined by
the surgeon as to be avoided. By utilizing an optimized planning algorithm to help
assist in navigation and operation of the surgery, it is the hope of the dissertation
author that these risks could be mitigated and/or eliminated. Path planning is a
merely geometric matter, because it is defined as the generation of a geometric path,
with no mention of any specified time law. On the other hand, trajectory planning
consists in assigning a time law to the geometric path. In most cases, path planning
precedes trajectory planning but is not necessarily a distinct phase of the surgical
procedure, see Gasparetto et al., (2015).
12
1.4.1 Robot 3D Path Planning Algorithms
In LaValle and Kuffner (2001), the authors developed Rapidly Exploring Ran-
dom Trees (RRTs). LaValle (2014) developed this algorithm to deal with high
dimensionality and the issues that can arise. In the same work, the investigators
found that RRT planning algorithms are optimally suited for kinodynamics.
What makes RRT unique as a path planning algorithm is that there are no
existing nodes to travel between and to help orient the algorithm Chin, (2019). The
robot is free to move about in the free space, (Cfree). To reach the goal state, random
points are generated and are connected to their nearest valid neighbor (Qnear), and
the pattern repeats randomly until the goal state is reached. A big caveat to this
is that there are registered obstacle spaces (Cobs). Each point, or vertex, has to be
checked by the algorithm in order to ensure that it is in the free space, and not
in the obstacle space. If it is in the obstacle space, the vertex is thrown out and
another random point is generated and that point is checked. If considered valid,
it is connected to its nearest neighbor. A couple of issues arise with this style of
planning algorithm: first, that the points are randomly assigned, and can cause
issues by either going into (Cobs) or coming too close to the obstacle space Chin,
(2019), completely at random, and can potentially have issues with regenerating
a discarded vertex if the program does not record a discarded vertex. Second,
the robot does not have existing nodes to travel between, as opposed to Dijkstra?s
algorithm with which ones uses a graph to resolve the issue. Third, the user must
determine how the shortest path will be chosen or determined, as RRT in its base
13
form does not automatically generate the shortest or smoothest path Chin, (2019).
A more detailed description for this PPA is discussed in Chapter 2.
1.4.2 Path Planning Algorithms and Cochlear Implantation
As robotic surgical tools advance in design, research groups have been working
on determining how to best perform the cochlear implantation surgery procedure
with a variety of path planning algorithms that sometimes include obstacle avoid-
ance, which is a focus of this dissertation. Pile et al., (2011) presented a method for
optimal motion planning for the insertion of electrode arrays. For a system with four
DOF, the authors used an objective function to set the optimal path by avoiding
the upper and lower bounds of the objective function, where the upper bound was
defined as contact with the outer wall of the scala tympani and the lower bound was
defined as contact with the inner wall.
Wanna et al., (2014), continued their work with optimal motion planning for
the insertion of electrode arrays into the scala tympani, with the addition of force
feedback, to further optimize how the electrode array can be inserted. Two sets of
experiments were conducted with this new modification utilizing plastic models of
the cochlea, and subsequently, cadaver cochleas. The results showed that signifi-
cantly less force was needed for the insertion of the electrode arrays into the cadaver
models, especially, with the use of admittance control law.
In the work of Zhang et al., (2009), the researchers created a customized
path planning algorithm based upon a weighted optimization function to determine
14
which of two different experimental arrangements helped reduce insertion forces for
cochlear implantation surgery. The goal was to generate the optimal path requiring
minimal amount of insertion forces needed for electrode insertion. They were able
to obtain a minimal insertion force of 3.9 g, which was an 18.8 % reduction in forces
from the maximum insertion forces that they used in these experiments.
Path planning in 3D environments has great potential for mapping out com-
plicated tasks and real world problems, see Chen et al., (2010). However, the dif-
ficulties increase exponentially with kinematic and dynamic constraints. Different
algorithms were generated to address to these new difficulties, including the sam-
pling based algorithms, node based optimal algorithms, mathematical model based
algorithms, bioinspired algorithms, and multi-fusion based algorithms, see Chen et
al., (2010). Important distinctions are made between path planning, optimal path
planning, and trajectory planning. Path planning refers to a continuous process
without break from an initial state to a goal state in the environment or world
space.
In the paper written by Aebischer et al., (2022), the authors focus on aspects
of robotic cochlear implantation (RCI) with regards to the insertion angles and their
effect on insertion forces and ease when inserting the implant into the cochlea. They
conducted motorized insertion of the electrode array into the scala tympani (part
of the cochlea) and determined the insertion forces, and their potential impact on
patient outcome. They concluded that having an insertion angle that was parallel (or
as feasibly close to parallel as possibly) provided the greatest reduction in insertion
forces. In the work of Panara et al., (2021), there is a review of the current state of
15
RCI implantation research, with which there has been success in the early stages of
development, minimizing facial nerve paralysis. However, the major challenge noted
by Panara et al., (2021), of the current research is to prove efficacy over traditional
approach in clinical trials. Novel methods for monitoring nerve neuromonitoring
schematic was discussed in the work, as well as schemes for dealing with thermal
injury prevention during the mastoiectomy Panara et al., (2021).
In the publication presented by Auinger et al., (2021), the authors look into
reducing scarring, recovery time, and to improving patient outcomes in RCI. They
also had a secondary objective of assessing the feasibility of the standard drill bit
size of 1.8 mm, and determined that a smaller drill bit of 1 mm to 1.7 mm drill bit
could improve feasibility of the drilling for keyhole access in RCI by up to 100%.
The authors found that the current slice thickness in CT scans or other DICOM
images presented issues in trajectory planning, and determined a slice thickness of
less than 0.3 mm would aid in the success of trajectory planning.
1.4.3 Summary of Previous Work
While the work of the authors cited in this review have made tremendous
advances in robotic cochlear implantation surgery, there are still some areas that
could be further developed. For example, in the work of Caversaccio et al., (2017)
there is not obstacle avoidance, and the path planning generally routes through
the more traditional paths that pose added risk to critical anatomy structures such
as the ossicles and cranial facial nerve, and the chorda tympani. Given that the
16
margin of error is approximately 1.2 mm distance from the cranial facial nerve for
their algorithm, it is recommended that one consider adding in obstacle avoidance
to ensure more safety for critical structures and find alternative paths for the drilling
process.
1.5 Clinical Information
The global population with severe hearing loss or congenital deafness is roughly
5 % of the world?s population, which is approximately 470 million people. Of that
population, 32 million are children (most under 15 years). According to the WHO,
only approximately 10 % of those affected by hearing loss have access to hearing
aids. Candidates for CIS are considered after all resources have been exhausted, see
McPhillips (2019b). In 2019 alone, there were 58,000 CIS pediatric cases worldwide
ASHA, (2019).
In pediatric and young patients, negative impacts of severe hearing loss can
manifest itself in the following consequences and ways: Spoken language develop-
ment is delayed and in many countries profound hearing loss can have an adverse
effect on the academic performance of children. There is a much higher unemploy-
ment rate among the hard of hearing, especially younger adults, which negatively
impacts the economic and social development of that person, and on a larger scale,
the surrounding communities McPhillips, (2019a).
There is significant impact in finding solutions to improving the quality of life.
The main benefit of CIS to adult and pediatric patients is to partially or almost
17
completely reverse the negative impacts caused by severe hearing loss. In particular
for the pediatric patients, there are large benefits in academic, economic, and social
settings, due to greatly improved speech-language skills, and ability to understand
what is said around them (Kliegman et al., 2007). Studies on the efficacy of multi-
channel cochlear implants in the pediatric population have reported postoperative
speech perception and speech production results in post-lingually deafened chil-
dren and in children with congenital or acquired pre-lingual deafness, see Hopkins
Health System (2019).
While it is currently recommended that the age of the patient be at least 12
months (Kliegman et al., 2007) for obtaining an implant, there have been studies
that show that obtaining implants earlier in age can result in a great improvement
in speech (Ricci et al., 2013). As children have a huge ability to recover speech if
treated early (Kliegman et al., 2007; Kliegman 2007; and McPhillips 2019b), there
could be great benefits for automating pediatric CIS, especially, with regards to
anatomical obstacle avoidance, and also possibly reduction of surgical time. As a
goal of the PPA, the patient specific data utilized will allow for customization of the
surgical procedure, and allow for a path that can be used to automatically avoid
critical anatomy. This has the potential to be incredibly beneficial for patients, and
shows how PPA has capacity to work in small surgical environments than would
occur for adult patients.
Figure 1.1 can be found at Hawkins (2019) and Figure 1.2 is provided by
of Health Communication and Liaison (2018).
18
1. Outer Ear = Auricle (Pinna), and External Auditory Canal
2. Middle Ear = Malleus, Incus, Stapes bones, Tympanic Membrane and Eu-
stachian tube.
3. Inner Ear = Cochlea and Vestibule, and Semicircular Canals (bony labyrinth).
Figure 1.1: Ear anatomy, adapted from Hawkins (2019).
1.5.1 Types of Cochlear Implantation Surgery
It is important to understand the different types of surgeries used to accomplish
CIS. Please note that the surgical methods for completing CIS are not limited to the
following procedures. These are cochleostomy and mastoidectomy (with posterior
tympanotomy approach), round window, and round window enlargement.
According to the work done by Richard et al., (2012) and (citeauthorMaj-
dani2009 et al., (2009), the cochlear implant is an approved therapeutic option for
patients with severe hearing loss. Since the Food and Drug Administration allowed
cochlear implantation surgery as a relief for severe deafness, multiple methods for
insertion have developed. Richard et al., (2012) reviewed several different types of
cochlear implantation surgeries: cochleostomy, round window enlargement (RWE)
19
Figure 1.2: Cochlear implant demonstrating different parts, adapted from NIH
(2018).
20
and round window procedure (RW). This was done in order to determine the amount
of trauma that results from each type of surgery, to determine the most effective pro-
cedure (produces successful cochlear implantation with minimal trauma to surgery
site). Based on the specific conditions in each case, the surgery style is selected.
As expected, the RWE style operation resulted in more abnormal tissue formation
than in the RWE method for electrode insertion, due to the nature of the surgi-
cal procedure Richard et al., (2012). As a result of this study (as well as others),
cochlear implantation has also been performed in a minimally invasive manner for
example, Majdani et al., (2009), and Caversaccio et al., (2017); the minimally inva-
sive surgical (MIS) procedure was specifically designed to produce significantly less
damage.
1.5.2 Steps to General Surgical Procedure
When developing a drilling PPA for CIS, it is important to understand what
steps are followed in the procedure, and what are the risks and complications. The
general overall steps of this surgical procedure are as follows Megerian, (2019).
General Steps for CIS: 1. Incision and Electrode array placement determined
2. Mastoidectomy and Posterior Tympanotomy
3. Cochlear Implant Receiver Well Drilled Out
4. Cochleostomy/ Round Window Insertion/ Round Window Expansion and Inser-
tion
5. Implant Tie Down and Electrode Insertion
21
6. Closure, and Radiograph
Generally, this surgery proceeds as follows: after anesthesia is administered,
pressure is placed over the mastoid tip to avoid inadvertently blocking of the cranio-
facial nerve Backous, (2014). The initial incision (behind the Pinna, along the
postauricular crease) is made after marking where the cochlear implant will lie in
the retromastoid region. As the surgeon begins, the surgeon prepares the mastoid
area and drills through the exposed mastoid part of the temporal bone, entering the
middle ear space through a posterior tympanotomy in order to expose the cochlea.
Once the mastoidectomy is completed, the round window is accessed and drilled
into, and then finally, the electrode array is recessed into the cochlea. The location
of the facial nerve is just posterior to the tympanotomy, is confirmed using electrical
stimulation. The goal is to approximate the location of the nerve without exposing
it, since it is very delicate, and the slightest insult can cause facial muscle paralysis.
Once located, drilling is resumed towards the round window, through which the
surgeon will reach the cochlea. Then, the electrode array is gently inserted, con-
nected to the external battery and audio tests begin, making sure that the implant
functions properly (radiograph tests), and then the surgical wound is closed.
1.6 Description of Problem
The goal of this section is to help translate the information gained from the
surgical setting to the path planning algorithm, specifically, for the drilling portion
22
of the surgical procedure so that it is understandable, and can be utilized for the
dissertation project. More specifically, the goal of the current work is to do with
bone drilling applications in ?small? space and the need for position control.
At an internship, the candidate has shadowed several cochlear implantation
surgeries to gain understanding of how the anatomy of the ear is structured and
the surgical procedure known as cochleostomy and mastoidectomy is performed.
Briefly, the surgeon prepares the area for drilling using incision and drills through
the exposed bone at a very slow rate, until the facial recess is visible. The surgeon
then tests the location of the cranial facial nerve by using electrical conduction.
The goal is to determine the approximate location of the nerve without exposing
it, since the nerve is very delicate, and the slightest damage can cause facial muscle
paralysis. Once the cranial facial nerve is located, drilling is resumed. The direction
of drilling is directed towards the round window, through which the surgeon will
reach the cochlea. Then, the electrode array is gently inserted, and connected to
the external battery and audio tests begin, making sure that the implant functions
properly, and then the surgical wound is repaired by stitching.
In order to accurately describe this surgical procedure for path planning algo-
rithms it is necessary to not only understand how the procedure is performed, but
to understand what the anatomy is, and why the surgeon takes the path that he or
she chooses. Additionally, it is important to understand the frame of reference from
which the tools are working in comparison to the anatomy, what will be labeled as
obstacles, what would be labeled as an initial state and a goal state, respectively,
and how to take this information into a path planning algorithm.
23
1.7 Outline of Dissertation
The rest of this dissertation is organized in the following manner. In Chap-
ter 2, the original model of the RRT path planning algorithm without obstacle
avoidance and collision detection will be discussed Poley and Balachandran, (2016).
The updated model of the path planning algorithm to include obstacle avoidance
and collision detection, along with a highly theoretical model of a cochlea is demon-
strated along with the results Poley and Balachandran, (2017). In Chapter 3, several
3D inner ear models developed from the CT scans belonging to several anonymous
pediatric cases of varying ages are shown along with the geometric modeling and re-
spective equations, the methods used, and the trocoidal models (3D representations
utilized for obstacle avoidance). In Chapter 4, the results of the image processing are
presented along with the updating and redesigning of the path planning algorithm
regarding the obstacle avoidance, the non-holonomic constraint conditions, and the
efficiency of the RRT algorithm itself. Additionally, the algorithm was updated to
include sequential quadratic programming for smoothing of the paths. In Chapter
5, the contributions following from this dissertation effort are summarized, along
with some thoughts for future work. Appendices on additional technical details and
references are provided at the end of this dissertation.
24
Chapter 2: RRT Kinematics, Dynamics and Obstacle Avoidance1
2.1 Overview
The work demonstrated in this chapter addresses a major research gap in
designing a successful robotic cochlear implantation surgery, specifically with an
algorithmic determination of drilling path with possible obstacle avoidance. The
focus of the work presented here is on implementing an optimized path planning
algorithm for the robot trajectory. The vast majority of the studies cited presented
in Chapter 1 do not address the robot trajectory or the cutting path (Chapter 4).
The models that were generated in this chapter are the result of fusing a math-
ematical model and the RRT algorithm in a 3D setting. Holonomic constraints
were considered for these studies, and basic obstacle avoidance models have been
constructed. Additionally, the issues and challenges that arose as this dissertation
work was undertaken are addressed further in Chapter 4. Specifically, in the work
addressed in Chapter 4, considers the addition of a cutting path for the surgical
procedure.
1This chapter is based on the work contained in the publications: Poley and Balachandran
(2017), Motion analysis of robot arm with movement restriction, in: ASME 2016 International
Mechanical Engineering Congress and Exposition, volume Volume 4A: Dynamics, Vibration, and
Control, pp. 1-9. and Poley and Balachandran (2016), Motion analysis of robot arm for obstacle
avoidance, in: ASME 2017 International Mechanical Engineering Congress and Exposition, volume
Volume 4A: Dynamics, Vibration, and Control, pp. 1-9.
25
Because of the previous work by Chen et al., (2010), the author generated
models using fusion of a mathematical model and the RRT algorithm in a 3D sys-
tem. The author?s model improves the field by the use of mathematical modeling,
avoidance of obstacles, and the use of a simulated cochlea.
2.2 Experimental and Computational Background
2.2.1 Accounting for Constraints
The mathematics for the motion of the robot manipulator arm are defined in
the joint space, as the intent was to control the movement of the robot arm along
its trajectory from the initial state to the goal state.
In the constraint derivation proposed here, a unique constraint has been used
so that one can restrict the manipulator arms based torques that would be applied
when using Euler-Lagrangian EOMs, or alternatively, generate torques for the sys-
tem kinematics (obtained from the Denavit Hartenberg Parameters) used for the
manipulator arm.
To take into account the constraint applied to this system, the author combines
Lagrange multipliers and discretized state space representation of the system for
joint variables path planning. By doing so, this allowed the system?s constraint
equation to be derived while utilizing the governing system of Eq. (2.1).
Dq? +C(q, q?)q? + g(q) = T (q) (2.1)
26
Here,D is the Inertia Matrix (n?n), C is the matrix of Coriolis and centripetal
terms (n?n), and g is the gravitational force vector (n?1) written as shown in Eq.
(2.2). Note that this equation is applied to every joint DOF of the system. Please
note that the qi terms are the joint coordinates, and are represented by ?i. ?i are
relative angles between the links Li and Li?1. The robot modeled in this work has
multiple joints per link, and there are joint DOF associated with each of them. The
author follows the notation set out in Spong et al., (2012), unless otherwise noted.
g(q) = (g1(q), ..., gn(q))
? (2.2)
The vector of coordinates is written as:
q = [q1, ..., q ]
?
n (2.3)
Please note that the generalized coordinates shown in Eq. (2.3) corresponds
to the values of ?i, which are relative angles between the links. Additionally, T is
defined as the torque input for the system. Eq. (2.1) can be rewritten in state space
form as shown in Eq. (2.4).
?? ?? ?? ???? ?? ?? ?? ?? ???q??? ??0 I= ????q?? ?? 0 ?? ?? 0 ???+ + ? (2.4)
q? 0 ?D?1C q? ?D?1g D?1
Regarding constraints, it is important to identify what constrains a system,
as most, if not all, robotics systems have constraints applied to them. Additionally,
understanding the patient specific definition of the constraint will help improve the
27
outcome of the current work as it allows for customized path planning algorithms
to be generated.
It is important to note the difference first between holonomic and non-holonomic
constraints, and then understand how both the equality and inequality constraints
factor into the current problem. These constraints all affect the movement of the
robotic system, and will need to be defined properly.
Holonomic constraints are defined as constraints that are integrable, while
non-holonomic constraints are not integrable (see Chapter 6 in Greenwood, 1997),
see differential form of the constraints shown in Eq. (2.5). Inequality constraints
are non-holonomic constraints, which are used in Chapter 4 of this dissertation.
However, it is important for the purpose of this work to determine whether or not
each constraint applied falls into a category of holonomic or non-holonomic on a
case by case basis.
The holonomic constraints on the system can be defined as shown in Eq. (2.5).
?n ?hj
dhj = dxi = 0 (2.5)
?x
i=1 i
Here, dhj is the differential of the j
th holonomic constraint applied to the
system. The holonomic constraint equations are used for the multiple links in each
system that is studied in this work.
Another constraint type to consider would be equality vs. inequality con-
straints. Equality and inequality constraints are given below in Eq. (2.6) and Eq.
(2.7), respectively.
28
zi(x) = 0, i = 1, ..., s (2.6)
zi(x) ? 0, i = 1, ..., r (2.7)
Eq. (2.1) can be rewritten in state space form as shown in Eq. (2.8). In order
to allow for an iterative quantification of each node Qnew as it is added to the tree,
the state space form of the EOM is spatially discretized, and shown in Eq. (2.8).
x? = f(x) +B? (2.8)
where f(x) has a size of 2n? 1, and B has a size of 2n? 2n, and ? has a size
of 2n? 1 Here, x the possible positions for the robotic manipulator arm LWR ? X
generated from the RRT path planning algorithm. x? has size 2n? 1, and x? has size
2n? 1. Furthermore:
?? ?
B = ??0n?n??? (2.9)
D?1
Recall that D has a size of n?n. T is the set of all torque input on the system,
and ? and is one set of the torque input applied to the system at each distinct point
in time, and is a function in time.
Additionally, B is the control input matrix applied to the system. Please note
that, in order to simplify the computations, the following form has been utilized.
Eq. (2.1) is rearranged into Eq. (2.8), and is written in matrix form, which is
29
shown below in Eq. (2.4). The values of x here are still the possible positions for
the LWR manipulator arm along the path generated by the RRT.
Eq. (2.8) is rewritten as Eq. (2.10).
xk+1 ? xk + x?kdt = xk + (f(xk) +B?)dt (2.10)
Eq. (2.10) is recast below as Eq. (2.11).
xk+1 ? xk
B? = ? f(xk) (2.11)
dt
Eq. (2.11) is designed to follow the pattern set out by the RRT Algorithm,
where xk is Qnear and xk+1 is Qnew. This helps determine how the inertia and the
torque are valued at each iterative step. Additionally, in looking at how this system
can incur both linear and nonlinear constraints, the application of nonlinear con-
strained optimization is highly beneficial to this system. Since anatomy generally
doesn?t fit into a simple circle or box equation, it is important to consider how to
best optimize the system in a manner that would adapt to a patient?s anatomy. The
method considered is nonlinear constrained optimization, for which the author sup-
plemented the RRT path planning algorithm presented in this chapter and replaced
it entirely in Chapter 4. The torque is minimized, thus leading to the following
constrained objective function:
min ||B? ||2 (2.12)
??T
30
subject to:
???
vi ? 0.05 rad/s??????
a ? 0.10 rad/s2????? for Constrained Optimization of Robot Path (2.13)i ???
Eq. (2.13) has a listing of the representative values, which are at the high end
of the considered velocity and acceleration ranges. These are applied to each DOF,
and are for the values of ??i and ??i. The Eq. (2.13) is valid for values of i = 1, ..., 7.
The linear velocity and acceleration limits listed in Eq. (2.13) are to be defined as
switch constraints on the system. These are set low to control the movement of the
system for the surgical procedure. The angular velocity and acceleration terms are
represented by the coordinates used to describe each joint on the robot. While the
velocity and acceleration terms are not necessarily kept constant, but will likely be
kept at very low speeds for the surgical procedure, the values for which would be
controlled by the surgeon for the purposes of this work. Information on restricting
the motion of the constraints is based on the information provided by Company,
(2019). These are the constraints that are applied to the path as the LWR goes
from the initial state to the goal state. Constraints applied to the LWR?s path will
take the matrix form, as shown below in Eq. (2.14).
Please note that ? ?= t. Time is represented by t, and the time step dt for all
the simulations was dt = 0.1 seconds.
31
J (x )x? = 0 (2.14)h k
Jh is the constraint Jacobian of the system, with a matrix size of p?n, which
holds if the equations are linear. No angular constraints would be applied to ?i on
the robot path for the LWR, the angular constraints are applied to the cutting path
demonstrated in Chapter 4 of this dissertation.
Now, in order to guarantee that the new states obtained from Eq. (2.10), give
the state closest to the Qrand values obtained from the RRT planning algorithm, it is
necessary to find T o, which is defined as the optimal torque. Note that xk = Qnear
and xk+1 = Qnew. With this understanding, one makes use of the following the
following partial derivative equations. Optimizing torque and the control input is
done to allow for smoother movement between the different nodes on the RRT path.
Since the optimal torque is not known, it is determined by the algorithm for each
case. Please note that ? has a size of n? p.
L(?, ?) = (B?)?(B?)? ??Jh(f +B?) (2.15)
?L
= 2B?(B?) +B?J?h? (2.16)??
?L
= Jh(f +B?) (2.17)
??
32
Please note the following, for the subsequent steps:
Inv(A)? = (Inv(A))? (2.18)
By taking the partial derivatives of Eq. (2.15), it is possible to solve for the
Lagrange multipliers ?, which is needed to help find the optimal torque input for
the system. Partial derivatives of Eq. (2.15) are taken with respect to T and ?.
Here, the partial derivatives are taken with respect to torque T and the Lagrange
multiplier ? so that the optimal torque can be found for the system as it is currently
constrained.
? = 2(Jh?BB
?J ?h? )
?1[Jh?B(B
?B)?1B? + Jh?f ] (2.19)
The optimal torque is found in Eq. (2.20).
? o = (B?B)?1B? ?B?J ? ?(J ?BB?J ? ?)?1h h h V (2.20)
where V = [Jh?B(B
?B)?1B? + Jh?f ] in equation (2.20).
As the candidate is looking to optimize PPA with the new found optimal
torque T o, the value of ? o is substituted in place of ? into Eq. (2.21), and the RRT
planning algorithm is updated and optimized according to this constraint.
Qnew = Qnear + [f(Q
o
near) +B? ]dt (2.21)
After setting the derivatives to zero for stationarity and substituting Eq. (2.16)
33
into Eq. (2.17), it is possible to first solve for ?, and then subsequently solve for ? o.
The author is looking to solve for the partial derivatives.
2.2.2 Initial States and Goal states for System
Table 2.1: Initial and Goal States for Two Link, 4 DOF System (Before constraints
are taken into effect).
Qi,g ?1 ??1 ?2 ??2 ?1 ??1 ?2 ??2
Qi 0.1 0.1 0.1 0 0.1 0.1 0.1 0
Qg 1 0.1 1 0 0.5 0.1 0.5 0
These points were set as initial test cases for the RRT PPA due to their
proximal location, and the variability in responses.
Table 2.2: Initial and Goal States for 7 Link, 7 DOF System (Before constraints are
taken into effect).
Qi,g ?1 ??1 ?2 ??2 ?3 ??3 ?4 ??4 ?5 ??5 ?6 ??6 ?7 ??7
Qi 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Qg 0 0 0 0 0 0 0 0 0 0 0 0 0 0
For simplification of the problem, at the initial stages of the presented work,
the scala tympani was modeled as a simplified Logarithmic Spiral, with model:
34
x(?) = aeb?
y(?) = a cos(?)eb?
(2.22)
z(?) = a sin(?)eb?
where r = aeb?, as demonstrated by Weisstein, (2019). It is mentioned that the
control input matrix B is different from the constant value b in this equation.
2.2.3 Generation of Path Planning Algorithms
As in the planned system, a robotic drill is to be used to perform CIS, it is
important to define the state space. From the motion planning perspective, the
state space is a set of possible transformations that could be applied to the robot,
which is commonly referred to as the configuration space LaValle, (2014).
When defining the workspace, C, it is important to properly define the free
space Cfree and the obstacle region Cobs LaValle, (2014). Both the free space and
the obstacle region belong to the entire workspace, C, which is contained in the
three-dimensional world space defined as W = R3. In W is the region defined as
the workspace, or C. It is in some portions of that environment that the robot link
i, written as Ai, is to operate. More specifically, when the Ai operation is solely
in the free space, then, one write as Cfree. The rest of the workspace is occupied
by the obstacle region, defined as Cobs. The definitions of these two regions of the
workspace can be written as shown here Cfree ? C and Cobs ? C, see example in
35
LaValle, (2014).
However, it is important to set up the proper configuration space for the 3D
environment that is necessary to properly test this RRT PPA. The equations found
in Chapters 5, 6, & 7 of LaValle, (2014) have been adapted here to work with a 3D
environment.
Cobs = {q ? C?A(qi) ? O} (2.23)
where:
qi = (?1, ??1, ?2, ??2, ?1, ??1, ?2, ??2) (2.24)
Eq. (2.23) defines the subset of the configuration space that contains all con-
figurations of the robot arm to collide with obstacles or to have a self-collision. The
goal is to remove these configurations defined in Cobs from the total possible config-
urations of the robot arm in the workspace. This will allow for the robot arm to
traverse an obstacle free and collision free path to the goal state. The remaining
robot arm configurations exist in Cfree.
LaValle, (2014) defines C as a topological space, and Cobs as a closed set,
and Cfree as an open set. As a result of this definition, the robot arm may come
arbitrarily close to the obstacle while remaining in Cfree. Please note that int means
?interior?.
If a link of the robot arm, defined as Ai touches an obstacle, O, which is a
subset of Cobs, then we can state in Eq. (2.25) that the interior of the obstacle region
and the interior of the robot link qi intersection in no allowable configurations.
36
int(O ? int(A(qi))) = ? and O ? int(A(qi)) ?= ? (2.25)
This allows us to remove any possible configurations that would result in se-
rious self collision or serious collision with obstacle regions.
2.3 Results
The goal of the RRT algorithm is to generation of a straight line path, and the
goal of the author is to minimize deviation from the path. The results are presented
for two different simulated robot manipulator arms: i) a two link manipulator arm
with 4 DOF (revolute joints only), and ii) a LWR system. For each manipulator
arm, different simulations were performed using the kinematics or the Lagrangian
Equations of Motion (EOM); however, for both cases the author utilized the unique
constraint equation in order to compute the trajectory between the initial state and
the goal state. The simulations were used to test the effectiveness of the constraint
equations in effecting the torque input to the system, and bring the robot states
to the desired angular positions and angular speeds. In these cases, there is no
obstacle. Before the simulations were started, the kinematics and dynamic EOMs
were studied by using Mathematica to visualize how the systems would be affected.
The applied torque in the simulation would be limited to factory recommended
restrictions. Please note that reaction forces were not considered in this particular
algorithm due to the simplified nature of the setup.
Due to the noisy nature of the randomized initial data, a low pass filter was
37
applied and a time step of 0.1 seconds was used to smooth the torques for the
manipulators, as well as for the angular position and angular speed plots (see Figures
2.2, and 2.3, 2.4 and 2.5). The filtering of the data to remove noise was necessary to
remove the jagged parts of the trajectory that would cause the system to undergo
unnecessary torque changes or angular displacements, while still remaining true to
the data obtained from the RRT path planning algorithm. The smoothed data
shown in the figures are used to develop trajectories in each case. The step size of
the RRT path planning algorithm is dQ = 0.1
[ ]
Qi,g = ? (2.26)1 ??1 ?2 ??2 ?1 ??1 ?2 ??2
2.3.1 Two Link Manipulator Arm (4 DOF) Case
The values for the initial state and the goal state were chosen at random
to determine the effectiveness of the unique constraint equation. In this case, the
author used the following initial conditions.
( ) ( )
Qi = 0.1 0.1 0.1 0 0.1 0.1 0.1 0 Qg = 1 0.1 1 0 0.5 0.1 0.5 0
(2.27)
The values of Qi and Qg are shown in Eq. (2.27), and it is important to note
that these values stayed the same for both of the cases presented for the two link
manipulator arm. Some additional parameters, which were specified to generate
38
the simulations were mass, gravity, length of the respective links, and control input.
The parameter values used for this manipulator arm are m1 = 1kg , m2 = 1kg,
L1 = 1m, L2 = 1m, g = 9.81m/s
2, and B(u) = 3.
The goal of generating this algorithm is to get the robot to follow a straight
line trajectory as seen below in Figure 2.1.
Figure 2.1: Desired robot trajectory for each link.
2.3.1.1 Kinematics
In the sub-plots shown in Figure 2.2 and Figure 2.4, the histories associated
with the states are shown. Furthermore, in the sub-plots of Figures 2.3 and 2.5, the
torque inputs associated with the corresponding degree of freedom are shown.
In Figures 2.2 and 2.3, the red colored data represent the original data gen-
erated for Qnew, which are very noisy. The thin blue lines in these figures are used
for the filtered data that were obtained after using a low pass filter to get rid of the
high-frequency noise. It is noted from Figure 2.2 that after initial transients in the
39
1. The red, noisy data are the complete possible positions generated with this
algorithm.
2. The thin blue lines are the angular positions for each joint as the LWR traverses
the final path.
Figure 2.2: Kinematics case: Angular positions and angular speeds for ?1, ??1, ?2,
??2, ?1, ??1, ?2, and ??2.
40
1. The red, noisy data are the complete possible positions generated with this
algorithm.
2. The thin blue lines are the angular positions for each joint as the LWR traverses
the final path.
Figure 2.3: Kinematics case: Torques associated with states ?1, ?2, ?1, and ?2.
41
angular positions, there are relatively not much variations. This can be said to be
true of the filtered angular speed data as well. In Figure 2.3, as mentioned earlier,
the torque inputs applied to each DOF of the two link manipulator arm are shown.
Again, relatively minor variations are only observed after the initial transients.
2.3.1.2 Dynamics
As in the previous section, in Figure 2.4, the angular positions of the two link
system as well as the corresponding angular speeds are illustrated. Furthermore, in
Figure 2.5, the torques applied to each DOF of the two link manipulator arm are
shown. As in the prior plots, the data with noise and the filtered data are shown.
Again, after initial jumps in the angular positions, as shown in Figure 2.4, there is
relatively not much variation. The trends in angular speed histories are similar. The
illustrations provided in Figure 2.5 for the torque inputs applied to each DOF of the
two link manipulator arm show that the inputs settle down after initial transients.
However, with consideration of dynamics, the variations in the torque inputs are
higher than that noted previously in Figure 2.3.
2.3.2 LWR Case
For this case, the values for the initial state and the goal state were chosen at
random to determine the effectiveness of the unique constraint equation. The initial
and goal states were both chosen to have the values listed in Eq. (2.3) and Eq.
(2.3). The parameter values were chosen to simplify the models, and they follow
42
1. The red, noisy data are the complete possible positions generated with this
algorithm.
2. The thin blue lines are the angular positions for each joint as the LWR traverses
the final path.
Figure 2.4: Dynamic case: Angular positions and angular speeds for ?1, ??1 , ?2 , ??2,
?1, ??1, ?2, and ??2.
43
1. The red, noisy data are the complete possible positions generated with this
algorithm.
2. The thin blue lines are the angular positions for each joint as the LWR traverses
the final path.
Figure 2.5: Dynamic case: Torques associated with states ?1, ?2, ?1, and ?2.
44
the values listed in Company, (2019) are values that are used in Chapter 4 of this
dissertation. m1 = m2 = m3 = m4 = m5 = m6 = m7 = 1kg, L1 = 0.3105m,
L2 = 0m, L3 = 0.4m, L4 = 0m, L5 = 0.39m, L6 = 0m, L7 = 0.078m, g = 9.81m/s
2,
and B = 3.
Table 2.3: Initial and Goal States for 7 Link, 7 DOF System (Before Constraints
Taken into Effect).
Qi,g ?1 ??1 ?2 ??2 ?3 ??3 ?4 ??4 ?5 ??5 ?6 ??6 ?7 ??7
Qi 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Qg 0 0 0 0 0 0 0 0 0 0 0 0 0 0
2.3.2.1 Kinematics for KUKA LWR IV+ (LWR)
Figure 2.6 the author shows the orientation of each angle, so that it is easier
to understand how the data from the robot trajectory figures are related to each
link. Understanding the direction of movement enacted on the robot gives a better
understanding of the framework for the algorithm. The RRT algorithm does not
naturally optimize, due to the nature of the algorithm LaValle, (2014).
In Figures 2.7, 2.8, and 2.9, the results obtained are shown for consideration
of kinematics with the unique constraints applied to the LWR. In each of these
figures, the filtered data sets show how the trajectory follows the path. Since the
path generated by the RRT is not a smooth path, the variations in angular position,
angular speed, and the torque reflect how the robot must move in order to follow
the non-optimal path. For the LWR case, it was found that only the simplest cases
could be solved with the program that has been created in this work.
45
Figure 2.6: KUKA DLR LWR IV+ Reference Diagram for Robot Trajectories:
Angles ?1, ?2, ?3, ?4, ?5, ?6, and ?7 are shown in proper orientation.
46
1. ?1 = black path
2. ?2 = dark blue path
3. ?3 = cyan path
4. ?4 = green path
5. ?5 = yellow path
6. ?6 = gray path
7. ?7 = pink path
Figure 2.7: Kinematics case: Angular positions for LWR associated with ?1, ?2, ?3,
?4, ?5, ?6, and ?7.
47
In comparing Figure 2.7 with Figure 2.2 and Figure 2.4, one can see that the
angular magnitudes for the positions of ?1 and ?2 are larger in the systems where
there is on 4 DOF, and smaller in Figure 2.7, which shows results for a 7 DOF
system. Similar results can be seen in comparing Figure 2.8 with the above 4 DOF
systems. Similar results are also seen in the results for the torques in both the
7 DOF system and the 4 DOF systems, reaching the conclusion that the angular
magnitudes decrease as the number of DOFs increase. However, it can be seen that
there is much more fluctuation in the results for the 7 DOF system, even in the
smaller ranges of angular position, angular velocity, and torque, suggesting greater
instability than in the 4 DOF systems. These results were focused on stabilizing
and smoothing the movement of the robot arm as it moved along the trajectory
generated for each system, which was the rationale for choosing to work with joint
space dynamics.
2.3.3 Rapidly-exploring Random Trees
The Rapidly-exploring Randomized Tree (RRT) program can be highly effec-
tive in a delicate environment such as the cochlea. RRT was first introduced by
LaValle, (1998). The planning algorithm was introduced as an efficient data struc-
ture and search algorithm for high dimensional spaces that can contain algebraic
constraints for the obstacles as well as differential constraints from the movement of
the robot LaValle and Kuffner, (2001) and LaValle, (2014). Despite these benefits,
there are some limitations to the planning algorithm, which require modifications
48
1. ??1 = black path
2. ??2 = dark blue path
3. ??3 = cyan path
4. ??4 = green path
5. ??5 = yellow path
6. ??6 = gray path
7. ??7 = pink path
Figure 2.8: Kinematics case: Angular speeds for LWR associated with ??1, ??2, ??3,
??4, ??5, ??6 and ??7.
49
1. ?1 = black path
2. ?2 = dark blue path
3. ?3 = cyan path
4. ?4 = green path
5. ?5 = yellow path
6. ?6 = gray path
7. ?7 = pink path
Figure 2.9: Kinematics case: Torques for LWR associated with states ?1, ?2, ?3, ?4,
?5, ?6, and ?7.
50
to optimize the system. In practice, as the number of degrees of freedom of the
system increase, the amount of time that the system takes to resolve the trajectory
exponentially increases, leaving much to be desired.
Figure 2.10 contains a summary of the advantages and disadvantages of work-
ing with this method before adapting the program for the mathematical model
presented in this chapter. The careful selection of the RRT algorithm was chosen
for the advantages listed in Figure 2.10. A Rapidly-exploring Random Tree (RRT)
is a data structure and algorithm that is designed for efficiently searching noncon-
vex high-dimensional spaces. RRTs are constructed incrementally in a way that
quickly reduces the expected distance of a randomly-chosen point to the tree. RRTs
are particularly suited for path planning problems that involve obstacles and dif-
ferential constraints (nonholonomic or kinodynamic). RRTs can be considered as
a technique for generating open-loop trajectories for nonlinear systems with state
constraints. An RRT can be intuitively considered as a Monte-Carlo biasing search
into Voronoi regions. Usually, an RRT alone is insufficient to solve a planning prob-
lem. Therefore, it can be considered as a component that can be incorporated into
the development of a variety of different planning algorithms LaValle, (2014). Fig-
ure 2.10 was generated by the author of this dissertation from data obtained from
LaValle, (2014) and Chin, (2019).
In Fig. 2.11 the author shows an example of an RRT path plan that is in
the middle of being developed. The blue dot represents qinitial that is the starting
point for the program. qgoal conversely, is the goal state of the program, and is
51
Figure 2.10: RRT Advantages and Disadvantages.
1. qinitial initial state
2. qgoal goal state
3. qnear nearest relative point on the RRT tree to Qg
4. qnew newest iteration added to the RRT tree
5. qrand iteration that may become part of the tree
Figure 2.11: RRT example.
52
the desired endpoint, and is represented by the green dot. The black dots are the
nodes that are generated when the RRT algorithm is proceeding from the initial
state to the goal state. qrand represents the random node that is generated. qrand
is uniformly sampled from the configuration space. Assuming that the connection
between qrand and the nearest node qnear is collision or obstacle free, and qrand is
within the distance that is limited by the step size, qrand will be added to the tree,
and then becomes qnew. When the next iteration of qrand is generated, one of the
most recent iterations, qnew will be designated as qnear when it is determined to be
the closest to the most recent of qrand. Eventually, this tree construction reaches the
goal state, after the system converges.
RRT or rapidly exploring random tree is an algorithm through which one
introduces an incremental sampling and searching approach that does not require
any parameter tuning LaValle,(2014). The exploration algorithm is considered to
be a ?greedy? one, and is ideal for sampling based motion planning LaValle,(2014).
The ?tree? (a number of paths from one point to another) is grown by connecting
a goal point to the nearest point in the state space. These paths are grown in an
iterative manner, which usually creates a dense tree structure. Bidirectional RRT
is similar to a singular directional RRT, except that in this case, one creates paths
going to and from each point LaValle,(2014). In Table 2.4, the pseudocode presented
in this dissertation is shown. This pseudocode was written based upon the RRT
pseudocode shown in the efforts of LaValle,(2014) and LaValle and Kuffner, (2001).
When defining the workspace, C, it is important to properly define the free
53
Table 2.4: RRT PLANNING ALGORITHM PSEUDOCODE.
RRT Algorithm
Define qG as Goal State
Define qi as Initial State
G.init
for N = 1 to k do
G. add vertex(qnew))
if qG ? ?= (G, qnew)
continue
qnear ? NEAREST(G,qnew)
G. add edge(qnear, qnew)
qnear relabeled qnear(i? 1) & qnew relabeled qnear(i)
Run Evaluate.Constraint(qnear(i))
Run Evaluate.Dist.to.GoalState(qG, qnear(i))
if qG ? qnear(i) then
Return G.
space Cfree and the obstacle region Cobs LaValle,(2014). Both the free space and the
obstacle region belong to the entire workspace, C, which is contained in the three
dimensional world space defined as W = R3. In W is the region defined as the
workspace, or C. It is in some portions of that environment that the robot link i,
written as Ai, operates. More specifically, when the Ai operation is solely in the free
space, then, one write as Cfree. The rest of the workspace is occupied by the obstacle
region, defined as Cobs. The definitions of these two regions of the workspace can be
written as shown here Cfree ? C and Cobs ? C LaValle,(2014). The algorithms 2.4
and 2.5 can both be used to generate the angular position for each rotation on the
robotic system. This can be plotted out once the angles are converted to Cartesian
representation.
The pseudocode shown in Table 2.5 is modified from the path planning al-
54
Table 2.5: RRT PLANNING ALGORITHM WITH OBSTACLE AVOIDANCE.
RRT Algorithm with Obstacle Avoidance & Collision Detection
Define qG as Goal State
Defineqi as Initial State
Define C
G.init
Define Cobs
for N = 1 to k do
G. add vertex(qnew))
if qG ? ?= (G, qnew)
continue
qnear ? NEAREST(G,qnew)
if Cobs ?= then ?= (G, qnew)
continue
qnear ? NEAREST(G,qnew)
G. add edge(qnear, qnew)
qnear relabeled qnear(i? 1) & qnew relabeled qnear(i)
Run Evaluate.Self.Collision.Detection (Ai, Aj)
if Self.Collision.Detection == False then
continue
Run Evaluate.EOM(qnear(i))
Run Evaluate.Constraint(qnear(i))
Run Evaluate.Dist.to.GoalState(qG, qnear(i))
if qG ? qnear(i) then
Return G.
55
gorithm developed from the authors? earlier work, see Poley and Balachandran,
(2017). In this updated planning pseudocode includes obstacle avoidance and colli-
sion detection to better simulate the cochlea. In adapting the pseudocode from the
authors? earlier work Poley and Balachandran, (2017), it was important to make
sure that the working environment was described in the pseudocode. Additionally,
for the purposes of this simulation, it was important to define the obstacle region
(Cobs) ahead of time. This way, the program evaluates whether or not a randomized
point (qrand) is in an obstacle region; if this is the case, it is discarded. Once this
was accomplished, the different constraints would be evaluated. The first constraint
to be evaluated is the equations of motion for the robot manipulator arm as done
previously by the authors Poley and Balachandran, (2017), and then a check would
be made to examine the trajectory relative to the geometric constraints used to
describe the cochlea to ensure that the trajectory did not collide with the obstacle.
This is defined by Eq. (2.25). If there is a collision, a few of the most recent nodes
on the tree are deleted, and the remaining data points are checked. Finally, if there
is no collision, and the system reaches the goal state, Qg, and the trajectory is found
and followed by the robot manipulator arm in the simulation.
Eq. (2.23) and Eq. (2.25) and presented next follow from earlier work LaValle,
(2014). Eq. (2.25) is used to describe the obstacle region. Eq. (2.25) is used to
inform the program to discard all intersections of any obstacle region with any of
the links of the robot, A whose motions are described by using the generalized
coordinates qi.
Please note that Eq. (2.23) and Eq. (2.25) were obtained from the book
56
Planning Algorithms by LaValle, (2014).
Here, since the robot manipulator arm has multiple links, the obstacle region
will not only include the cochlea?s walls, but will also include regions to avoid self-
collision between multiple links of the LWR. To address this, as a preliminary step,
collision was prevented by restricting the rotational degree of motion that each link
was allowed to move.
2.4 Cochlea Geometric Constraints as Obstacles
As can be seen in Figure 1.1, there are three general sections to the ear.
Hawkins (2019) generated anatomically accurate drawings of the cochlea as seen in
Figure 1.1. Since this is where hearing takes place, the electrode array is placed
into the cochlea, if a patient suffers from severe loss of hearing or deafness and is
a candidate for the cochlear implantation surgery (CIS). In an adult patient, the
cochlea is about 9 mm in diameter, 5 mm in height, 30 mm in length, and 2 mm
at its widest point (Sampson et al., 2007). This geometrical aspects mean that one
has a carry out a highly restricted surgical procedure, and for pediatric patients,
the procedure is even more restrictive.
As image guided software was not used for this work, an equation was used to
approximate the shape of the cochlea. Eq. (2.22) is a logarithmic spiral that has
been chosen to represent the cochlea as it had a very similar pattern. The logarithmic
spiral equation was obtained from Weisstein, (2019). With slight modifications for
orientation and the values of the arbitrary constants a and b, a model fairly similar
57
Table 2.6: Logarithmic spiral equation with arbitrary constant values.
i ai bi
1. 0.55 -0.12
2. 0.8 -0.06
3. 0.7 -0.11
4. 0.65 -0.08
5. 0.4 -0.1
6. 0.45 -0.07
7. 0.6 -0.09
Table 2.7: Average dimensions of cochlea in adult humans.
Diameter 9 mm
Height 5 mm
Length 30 mm
Widest Point 2 mm
in shape and design to the cochlea, as seen in Figure 2.12 was produced. The values
of the arbitrary constants shown in Table 2.6 were determined through numerical
experiments to generate an image that is as close to the cochlea shape as possible.
The values of a and b in Eq. (2.22) are constants.
2.4.1 Two Links, Four DOF Case
Anatomical dimensions of adults were initially represented in Table 2.7, as
it was not easy to obtain numeric data on pediatric cases. In the case presented
here, the author shows the motions of a representative two links, four DOF system,
as it is maneuvered around a variety of obstacles in its environment. The simula-
tions have been carried out using MATLAB. While the manipulator arms are not
representative of any physical robot system, they were chosen for the studies, for
58
Figure 2.12: Simulated cochlea for RRT studies with obstacle avoidance.
proof of concept. The earlier work of the authors Poley and Balachandran, (2017)
serves as the basis for the work presented in this dissertation. In the prior work, a
motion based constraint was used to keep the robotic system inputs at an optimal
torque value while moving from one location to the next on the trajectory Poley
and Balachandran, (2017).
The novelty of the current work is derived from the following aspects. Firstly,
the authors use a probability based path planning algorithm with constraints that
include the equations of motion of the considered system. Secondly, in addition
to movement restrictions that require the use of the optimal level of torque inputs
for multi-link robots while traversing from the initial state to the goal state, the
authors intend for the system to avoid obstacles. Once the image of the cochlea
was generated, it was added to the program as an obstacle. The dimensions listed
59
1. Green ? is initial state Qi
2. Yellow ? is goal state Qg
3. Red dots are vertices that represent possible locations for the robot
4. gray lines, edges connecting vertices
5. blue line final path generated by RRT
Figure 2.13: Projection of state space plot in ?1-??1 Space.
60
1. Green ? is initial state Qi
2. Yellow ? is goal state Qg
3. Red dots are vertices that represent possible locations for the robot
4. gray lines, edges connecting vertices
5. blue line final path generated by RRT
Figure 2.14: Projection of state space plot in ?2 - ??2 space.
61
1. Green ? is initial state Qi
2. Yellow ? is goal state Qg
3. Red dots are vertices that represent possible locations for the robot
4. gray lines, edges connecting vertices
5. blue line final path generated by RRT
Figure 2.15: Projection of state space plot in ?1-??1 space.
62
1. Green ? is initial state Qi
2. Yellow ? is goal state Qg
3. Red dots are vertices that represent possible locations for the robot
4. gray lines, edges connecting vertices
5. blue line final path generated by RRT
Figure 2.16: Projection of state space plot in ?2-??2 space.
63
in Table 2.7 were obtained from Hawkins, (2019) and (Sampson et al., 2007) as
averages for the dimensions of the cochlea in an adult human. These dimensions
were used to give a sense of proportion to the object that was generated. The goal
of this obstacle avoidance is to enter into the middle of the cochlea without the
robot manipulator arms touching the walls of the cochlea. The goal state is placed
equidistant between the two walls of the cochlea. As the system is expanded to
include more degrees of freedom, the goal state will be placed farther inside the
cochlea, and the robot will work to go through the round window entrance to the
cochlea. As the cochlea that is simulated is situated around the goal state, it makes
sense that with the modified RRT one would spend some time manipulating itself
around to process the best path solution.
For the two links, four DOF system, there are eight states:
qi = (?1, ??1, ?2, ??2, ?1, ??1, ?2, ??2) (2.28)
These simulations have been performed with N = 100,000 iterations. The
number of N is substantially high as the path planning algorithm generated for this
work searches the three-dimensional work space C for all possible positions.
In Figures 2.13, 2.14, 2.15, and 2.16, the unfiltered data from the simulation
are shown. These figures represent the information processed from a two link, four
DOF system. The red dots are the locations of qnew and qnear for the RRT tree.
The blue circles and lines form the trajectory going from the initial state (Qi)
(green ?) to the goal state (Qg) (yellow ?), or towards it with a defined converging
64
distance. The state space projections shown in Figures 2.13, 2.14, 2.15, and 2.16
are beneficial to examine if the system was able to successfully navigate a three-
dimensional environment with an obstacle and still come to the goal state for each
chosen state qi. From each of these plots, it is discernible that there is a tendency
for the path planning algorithm to search closer towards the goal state, which is
understandable, given that the obstacle is surrounding the goal state.
In Figure 2.17, the time histories of each degree of freedom are shown. The
red lines in each subplot indicate the raw unfiltered data from the system. The
black, cyan, yellow, and gray lines are the smoothed path plans that the robot
manipulator arm follows going from the initial state to goal state. The smoothed
path is indicative of how well the robot is able to maneuver in the work space and
around the obstacle region. In each of the angular position plots, it is evident that
there is a large transient in going from the initial state to the subsequent state,
before the robot manipulator arm position was able to converge in the position
state. Comparing the response history plots, it is discernible that in Figure 2.17,
specifically in plots a) and c), there are more variations in the movement before the
system levels off in the corresponding position state.
In Figure 2.18 and Figure 2.19, the time histories of the angular speeds are
shown with the degrees of freedom shown in Figure 2.17. The red data are the
unfiltered data from the RRT program?s randomization process, and the filtered
data resulting for a smoothed path are shown by using blue, green, orange, and
yellow lines in plots a) to d). It is interesting that the angular speed variations are
65
1. ?1 = black path for ?1
2. ?2 = cyan path for ?2
3. ?1 = yellow path for ?1
4. ?2 = light gray path for ?2
5. Red dots are vertices that represent possible locations for the robot
Figure 2.17: Angular position histories For ?1, ?2, ?1 and ?2.
66
1. The dark blue line represents the velocity utilized over the smoothed path to
go from Qi to Qg for ??1. item The green line represents the velocity utilized
over the smoothed path to go from Qi to Qg for ??2.
2. Red dots are vertices that represent possible locations for the robot
Figure 2.18: Angular speed histories for ??1 and ??2.
contained well by the torque inputs to the system.
The work presented in (Majdani et al., 2009) and Caversaccio et al., (2017)
provides a good basis for comparison with the future work of this research. There are
several important distinctions to note between earlier work Caversaccio et al., (2017)
and the work presented in this dissertation. Firstly, it is important to note that in
the earlier work Caversaccio et al., (2017), the researchers used image segmentation
and reconstruction of the image to register critical anatomy. Secondly, in previous
work they were able to construct their image segmentation through the utilization
of Computed Tomography (CT) images. By comparison, in this dissertation, a
mathematical model based on geometric approximation is used. The robot drill
used by Caversaccio et al., (2017) is simply meant to create the path and the system
67
1. The red, noisy data are the complete possible positions generated with this
algorithm.
2. The orange line represents the velocity utilized over the smoothed path to go
from Qi to Qg for ??1. item The yellow line represents the velocity utilized
over the smoothed path to go from Qi to Qg for ??2.
Figure 2.19: Angular speed histories for ??1 and ??2.
68
checks to ensure that the drill did not come too close to any critical anatomy. The
surgeons then place the cochlear implant, whereas in the present work, the authors
present a simplified simulation.
2.5 Remarks
Here, the author has developed path planning algorithms for planning of robot
arm motions in the presence of restrictions. From the results presented here, it is
evident that the constraints can be incorporated for the cases considered, in which
the manipulator arms were able to go from the initial state to the goal state while
being subjected to the system torques applied to each DOF. Information was also
provided on the smoothed torque, to illustrate how much torque is needed to get
from the initial state to the goal state. Comparisons between the manipulator arms
will be shown and analyzed, when the coding for the LWR?s dynamic simulations
is completed. For the future, multi-directional RRT trees are suggested for consid-
eration, as well as obstacles (cluttered environment) to better anticipate the needs
of both the surgeon and improve the precision of the surgery while operating. The
intention of using joint space dynamics for the robot trajectory was to create as
smooth a path as possible when working with a path planning algorithm in a 3D
environment.
It is clear that the RRT in its current status that there is considerable error
between the desired path and the actual path generated, and that considerable
work needs to be done to generate a path consistent with the desired trajectory.
69
In determining the desired path, considerable work went in to looking at filters to
attempt to reduce the nominal error.
This research has helped to extend the path planning algorithm of the author?s
earlier work to include obstacle avoidance. As an illustrative application for the use
of this algorithm, cochlear implantation surgery is considered. The results obtained
in preliminary work are encouraging and illustrative of the promise of the RRT
based approach. In order to successfully study cochlear implantation surgery is to
look at many aspects as possible, including the robot trajectory.
70
Chapter 3: Image Processing and Segmentation and Observation of
Surgical Procedures for Implementation in PPA
3.1 Overview
Image analysis is carried out on the pediatric surgical cases that are presented
in this chapter. This is done to help set up the anatomically correct environment
for the path planning algorithms generated and demonstrated in Chapter 4.
Several Image Processing programs were considered carefully in order to de-
termine the one that would best suit the needs of the dissertation. In the end, it was
determined that 3D Slicer would be the best fit program for a variety of capabilities,
including features that more easily facilitated the importing of the rendered segmen-
tation into MATLAB. The goal of the work presented in this chapter is to facilitate
a process for image processed anatomy to be utilized in path planning algorithms,
and to hopefully in the future offer this procedure to patients that were previously
excluded due to the severity of the congenital abnormalities, such as C.H.A.R.G.E
patients.
Additionally, a IRB Human Subject Research Determination (HSRD) docu-
mentation was filed, and the methodology utilized in this work was found to not
71
need an IRB, due to the lack of patient contact and the anonymous nature of the
temporal bone CT Scans. In this dissertation, the information obtained from anony-
mous CT scans are generated into 3D segmented models, which are then imported
into MATLAB for generating surface equations and subsequently used in the PPA
demonstrated. By generating surface fitting equations for the models presented in
this chapter, it is possible to compare the data and images with the results pre-
sented by other researchers. This is done to ensure accuracy for the models before
the models are integrated for a cutting path planning algorithm. The segmented
models will be used directly for the cutting path planning algorithm presented in
Chapter 4.
3.2 Literature Review
In the work presented by Pietsch et al., (2018), the authors show a close
correspondence to the curve fitting generated here, with a goodness of fit measure
R2= 0.95 for their fourth order polynomial equations, which are generated from
models of the adult cochlea and semicircular canals. In related work carried out
by Biferale et al., (1996), the authors show a moderate difference in the equations
constructed generated for the adult semicircular canals. Representations of these
semicircular canals have also been generated by using a Fourier Series Transform
Backous, (2014); Bradshaw et al., (2009).
McRackan et al., (2012) conducted a study comparing the dimensions and
geometric relationship of the anatomy and physiology of the inner, middle and
72
outer ear of children and adults. The findings of McRackan et al., (2012) showed
significant differences in a majority of the anatomy. A more acute angle between
the long axis of the basal turn (BT) of the cochlea and the lateral point of the EAS
was determined, as well as an acute angle between the long axis of the basal turn of
the cochlea and the plane of the facial recess. McRackan et al., (2012) determined
that in comparison with pediatric cases, the drilling trajectory in adult cases was
significantly better at maintaining a parallel trajectory to the EAC. The trajectory
of the drilling path must be adapted to the pediatric anatomy, and cannot be based
solely on the anatomy and drilling trajectory of adult CIS cases.
It is expected that the image processing can allow for anatomically correct
images of patient specific anatomy to customize the surgical procedure to each indi-
vidual patient. This is meant to help the dissertation author develop a better path
planning algorithm that can more easily facilitate the planning of CIS by having
every bit of information possible. This helps improve the accuracy of the cutting
path planning algorithm and potentially reduce the amount of bone that is removed
during the surgical procedure, and possibly reduce the amount of time that could
be spent in the procedure as well. Data points presented in this work have been
validated across several different programs and with multiple tests, and multiple
models have been used to generate statistically significant results.
A lot of the data points generated by Pietsch et al., (2018) is the most current
one to date for adult models. Related studies include those of Bradshaw et al.,
(2009), Biferale et al., (1996). Labadie et al., (2014) utilized image processing based
on CT segmentation to test on trial CIS.
73
3.2.1 C.H.A.R.G.E. Patients
Finally, C.H.A.R.G.E. patients, a specific subgroup of patients whose congeni-
tal condition includes profound deafness, are patients that have serious abnormalities
to the structure of the ear due to congenital issues. Due to the sometimes severe
nature of the abnormalities, it can be difficult for a surgeon to conduct the surgery,
and the typical ?landmarks? are altered or missing. Ricci et al., (2013) noted that
while their new method of working with C.H.A.R.G.E patients for CIS was more
helpful in the implantation, that a lot of the natural landmarks that surgeons use
for CIS are not indicated on a CT or MRI, which can lead to increased risk for
complications, causing some of these patients to be generally considered inoperable.
Working with CT scans of C.H.A.R.G.E patients will be one of the recommended
future work items in Chapter 5, as temporal bone CT scans of patients with this con-
dition are not currently available to the dissertation author. A goal of future work
beyond this dissertation would be to be able to make patients who are previously
considered inoperable, now viable candidates for surgery.
3.3 Importance of Anatomical Accuracy
The motivation behind having anatomically correct images of the inner ear,
and a better understanding of how the procedure worked, is to know how to best
implement path planning algorithms for convenient use in cochlear implantation
surgery. It is believed that utilizing the anatomically correct and patient dependent
models will be useful in customizing the surgical procedures. One of the benefits of
74
optimizing the path planning algorithm to prevent delicate features from potentially
becoming damaged in the surgical procedure. Additionally in future work, it may
be possible to produce an entirely new path that can mitigate the risk entirely, due
to the influence of the path planning algorithms.
The benefit of introducing a path planning algorithm for robotic cochlear im-
plantation surgery is that it can prevent risks due to nicks and tears in a tight
environment, and furthermore assist the surgeon by highlighting any unknown ab-
normalities that would be recognized as obstacles. A possible additional benefit
is that it can act as a training tool for surgeons who are newer and/or gaining
experience in surgery.
In the current work, the information obtained from anonymous temporal bone
CT scans are used to generate 3D segmented models, which are then imported
into MATLAB for generating surface equations and subsequently used in the PPA
demonstrations. The segmentation models resulting from the temporal bone CT
scans are labeled: sample No. 1, sample No. 2, sample No. 3, sample No. 4, and
sample No. 5.
The dissertation author makes comparisons between these simulated results
and our numerical and simulated results based on efficient algorithms Pietsch et al.,
(2018) and Biferale et al., (1996) to validate the accuracy of the methodology for
generating the surface equations for the cochlea.
75
3.4 Methodology
3.4.1 Segmentation of Inner Ear
In order to determine the necessary surface fitting equations and compare them
to models that have been derived in the literature mentioned above, it is necessary
to be able to generate 3D models. In this research, temporal bone CT images have
been used, as they were utilized for generating 3D models of the inner and middle
ear. For the purposes of this dissertation, all collected data points are anonymous,
these CT scans were done months to years prior to the work of the dissertation, and
no patient personal information was shared with the author for this work. All 3D
models generated from the CT scans will be referred to as a sample. Representative
images are show in in Figures 3.2 to 3.4.
Figure 3.1: General flow for 3D segmentation process.
Figure 3.1 shows the general process for generating and analyzing patient
specific anatomy of the inner and middle ear.
3D Slicer was the environment utilized for the segmented models generated
76
Figure 3.2: Axial view (red slice) of sample No. 3 of CT scan of temporal bone on
the left side of a patient.
Figure 3.3: Coronal view (green slice) of sample No. 3 of CT scan of temporal
bone on the left side of a patient.
77
Figure 3.4: Sagittal view (yellow slice) of sample No.3 of CT scan of temporal
bone on the left side of a patient.
Figure 3.5: Representation of Slicer environment.
78
based on CT scan. This environment will be further utilized in the next chapter, for
the generated PPA. From the data points obtained in this work, one can note the
anatomical variability from one case to another. Segmenting cases with a variability
in anatomical features is expected to be beneficial for constructing a PPA to work
over a wider candidate bases, as a library of anatomical structures is generated.
Figure 3.6: Segmented model of sample No 1. cochlea and round window, bony
labyrinth showing abnormal anatomical formations.
Figure 3.6, the author shows the cochlea (yellow) and round window (green)
generated from a temporal bone CT scan. While the bony labyrinth is not directly
affected by CIS, segmented models from samples such as this one are beneficial to
improving the cutting PPA. The length of the cochlea was determined to be 13.573
mm from the apex of the semicircular canals to the top of the cochlea, and the
79
width was determined to be 2.494 mm, at the widest point of the vestibule, which is
shown in Figure B.9. The cochlea itself was approximately 7.3 mm from the base of
the cochlea to the start of the basal turn, also represented in Figure 3.7. In sample
No.2, the total length of the cochlea from the apex of the semicircular canals to
the base of the cochlea was found to be 14.8 3 mm, and the width was found to
be 2.263 mm. In sample No.3, the total length of the cochlea from the apex of the
semicircular canals to the base of the cochlea was found to be 16.77 mm, and the
width was found to be 2.814 mm. In sample No.4, the total length of the cochlea
from the apex of the semicircular canals to the base of the cochlea was found to be
16.52 mm, and the width was found to be 2.92 mm. In sample No.5, the total length
of the cochlea from the apex of the semicircular canals to the base of the cochlea
was found to be 15.39 mm, and the width was found to be 3.87 mm. The mean
and standard deviation for this data is listed in Table 3.1. Please also see additional
figures in Appendix B.
The average length of the cochlea was found to be 6.87 mm in the patient
samples, measuring from the beginning of basal turn to the other end.
The segmented models that were generated in this work demonstrate notable
anatomical variability as candidates for CIS. This can be seen in Figures 3.6 and
3.8, as well as from the generated models in Appendix B. Segmenting cases with a
variability in anatomical features will be beneficial in getting the PPA to work over
a wider candidate base, as a library of anatomical structures is generated.
In Figure 3.8 the cochlea drawn in the inner ear segmentation model is hollow,
to create a more realistic model, for future plastic model experimentation testing.
80
Figure 3.7: Landmarks and anatomy of a normal cochlea Gray, (2022).
Figure 3.8: View of the cranial facial nerve and the round window generated from
sample No. 3. The different viewpoints of the inner ear models to show the
complexity of the delicate anatomy and the tight environment faced by surgeons.
81
Material Properties such as tensile strength will need to be compared to human
bone to determine how to properly scale the size of the plastic models. The full
scale temporal bone models will be a better approximation of the realities of the
procedure during experimental testing.
3.4.1.1 Proposed Patient Specific Drilling (Cutting) Paths
The necessity for proposing a patient specific cutting path can be seen in the
differences in the abnormalities shown in the CT scans from the patients below.
McRackan et al., (2012), has noted how surgeons have often felt that performing
CIS on children can be much more difficult than performing CIS on adults, due to
anatomical differences in the critical structures of the middle ear and other relevant
anatomy, especially when considering the variety of malformations that can occur
Hang et al., (2012). Not only is there a difference in overall size of the cochlea (7 -
15 mm in children and adolescents vs 30 - 40 mm in adults) McRackan et al., (2012).
Given the findings presented by McRackan et al., (2012) regarding orientation
of the cochlea, as well as the orientation paths cranial facial nerve in pediatric
candidates, the area for feasible drilling can be even tinier than first anticipated.
For this reason, it is important to see the 3D structures of the relevant anatomy.
Each patient?s unique anatomy slightly alters the proposed drilling path, in
addition it is important to note that the drilling path would move in 3 dimensions,
and that the above figures are merely proposed trajectories. It is important to note
that the orientation of the cochlea, the location of the cranial facial nerve in the
82
Figure 3.9: Slice from CT scan for patient 1, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed 2D representation of the
cutting path.
CT scan, as well as the orientation of the ossicles determine will be major factors
in determining the drilling (cutting) path in order to reach the round window (goal
state).
3.4.2 Methodology for Generating Surface Fitting Equations in MAT-
LAB
The reader is referred to Appendix B for the MATLAB codes used to generate
the surface fitting equations and correlating figures. The MATLAB Curve Fitting
Toolbox was used to analyze the data points and generate the cochlea surface fitting
equations.
1. Start by downloading the NRRDReader Program from Mathworks. These
83
Figure 3.10: Slice from CT scan for patient 2, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed 2D
representation cutting path.
84
Figure 3.11: Slice from CT scan for patient 3, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed 2D
representation cutting path.
85
Figure 3.12: Slice from CT scan for patient 4, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed 2D
representation cutting path.
Figure 3.13: Slice from CT scan for patient 5, axial view, showing location of
cochlea, ossicles, and cranial facial nerve, and a proposed a proposed 2D
representation cutting path.
86
generated binary tensors of data points for each slice of the segmentation imported.
2. For the purposes of this chapter, the 3D model imported to MATLAB on
an individual model basis (cochlea, round window, tympanic membrane).
3. However, before importing the files from slicer to be read, some ?pre-
processing? work needs to be done in order for the binary data points of the structure
to be readable by MATLAB, once done the data points from each slice in the CT
scan will fill a .txt file to be used for generating the surface structures.
4. Import the binary data points, formatting it and removing data points sets
to minimize workflow, and generate plots to check for consistency.
5. This data set is then concatenated so that the Curve Fitting Toolbox can
read it and generate results. These equations and data points points will be com-
pared to current literature for accuracy, and utilized in the Cutting path planning
algorithm to generate recognized obstacles.
Figure 3.14: General flow for surface fitting.
Figure 3.14 contains the overall concept for generating 3D models and then
obtaining the surface fitting equations. Importing the CT scans into MATLAB
is feasible, and this was tested out in the current work, but it was not able to
87
segment the specific inner ear models. However, after reviewing research literature
and analysis on the curve fitting toolboxes, as well as other MATLAB programs
on how to generate these equations, the author realized that the break-point in the
original segmentation data points was where the author needed to cut to analyze
the surface fitting equations.
3.4.3 Body-Centric Frame to World Frame
The axes are called R- Right, A-Anterior and S- Saggital as shown in Figure
3.15. For CT scans, this body-centric coordinate axis is either fixed the subject who
is being scanned, or the scanning table John,(2019).
Figure 3.15: RAS Coordinate System is a body-centric coordinate frame utilized
by CT imaging systems.
Relating the two coordinate frames can be done in a straightforward manner
through a homogeneous transformation Asada and Slotine,(1986), shown below in
Eq. (3.2). The homogeneous transformation matrix used to go from the body-
88
centric reference frame RAS to the world frame XYZ is shown below, and follows
the notation of Spong et al., (2012). This rigid-body transformation allows for the
change of reference frames from the body-centric reference frame RAS to the world
frame XYZ, which is used by the robot and the PPAs. In Eq. (3.2), it is important
to note that di represents the distance translated along the zi-axis. The angle ?i
rotates counterclockwise about the zi-axis. The distance bi?1 translates along the
xi?1-axis, and finally the angle ?i?1 rotates counterclockwise about the xi?1-axis.
H = Rotx,?Transx,bTransz,dRotz,d (3.1)
?? ??? cos(? ) sin(? ) 0 b????
i i ???
cos(?i) sin(?i) cos(?i) cos(?i) ? sin(?i) ?d sin(?i)? ?
?
H =
????
??
sin(?i) sin(?i) sin(?i) cos(?i) cos(?i) d cos(? ) ???? (3.2)i
??? 0 0 0 1 ????
In Figure 3.16, the point cloud that was imported from Slicer was validated,
and would later be used to generate surface fitting equations.
As can be seen from Figure B.1, the confidence is R2 = 0.9641, and the 4th
Order Polynomial generated to fit the semicircular canals fits very well. The data
points excluded from that set turned out to be from the lower part of the cochlea,
where the surface equations tend to be like a Fibonacci Logarithmic Spiral Pietsch
et al., (2018).
89
Figure 3.16: Cochlea and semicircular canals shape validated in point cloud plot in
MATLAB.
90
3.4.4 Dimensions of cochlea
In order to accurately determine the needed obstacle avoidance size, dimen-
sions must be measured.
The values obtained and presented in Table 3.1 is generated from 5 sets of
anonymous data points, and the temporal bone models illustrated later in this chap-
ter are from the same sets. All dimensions found in this dissertation were taken from
the segmented models, from importing them into Solidworks and measuring. The
original figures with dimensions can be found in Appendix B.
Dimensions generated from rendered CT scans for all data sets are in millime-
ters. Lc is the length of the apex of the cochlea to the top of the superior semicircular
canal. Wc is the width of the cochlea in the widest point of the vestibule. DRWc
is the distance measured from the midpoint of the round window membrane to the
midpoint of the first turn in the cochlea. DCT1E is the exterior diameter of the first
turn of the cochlea. DRW is the diameter of the round window.
These dimensions were measured to help create the obstacle regions in the
PPA, and also to define constraints and boundary conditions for the PPA discussed
in Chapter 4. The external diameter of the cochlea at the first turn was measured
since CIS can be performed as a cochleostomy (see Appendix A for further details).
The diameter of the round window membrane was measured since with CIS, one
often uses the round window membrane to insert the electrode array. In this prior
work, the models generated for that work represented solely the semicircular canals
for the viable cases. A Fourier series representation was generated to map out the
91
surface fitting on their samples of the semicircular canals.
Table 3.1: Dimensions for Boundary Conditions on cochlea from patient data sets
Sample No. Lc mm Wc mm DRWc mm DCT1E mm DRW mm
Sample No. 1 13.573 2.494 4.801 1.765 2.35
Sample No. 2 14.83 2.263 4.48 2.564 1.21
Sample No. 3 16.77 2.814 5.569 1.992 2.13
Sample No. 4 16.52 2.92 4.29 1.72 1.83
Sample No. 5 15.39 3.87 4.18 3.35 1.494
Mean ? 15.4166 2.8722 4.664 2.2782 1.8028
Std. Dev ? 1.1655 0.5505 0.499155687 0.6143 0.4132
There is reasonable consistency in the standard deviation values. The lowest
? value that was found for the distance, was from the diameter of the round window
membrane. This standard deviation had a value of ?D = 0.413. The largestRW
range of standard deviation was presented by the length of the cochlea, (?L =C
1.1655). While the cochlea overall shape and length is dependent on the age of the
patient, the author of this dissertation did not have access to that information for
the samples provided by Robbins,(2019). In comparing these models to the work
presented by Bradshaw et al., (2009), there are several important distinctions to
note: firstly, the models generated by Bradshaw et al., (2009) were generated for
an adult population and in an entirely different manner. The models generated for
that work represented solely the semicircular canals for their viable cases. Bradshaw
et al., (2009) generated a Fourier series equation to map out the surface fitting on
their samples of the semicircular canals. In this prior work, the models generated
for that work represented solely the semicircular canals for viable cases. A Fourier
92
series representation was generated to map out the surface fitting on their samples
of the semicircular canals.
3.4.5 Temporal Bone CT scan generation
Figure 3.17: Temporal bone of sample No. 3 showing the orientation of the
anatomy during the surgical procedure, along with a marker showing the location
for the initial state Qi located on the temporal bone.
Please note that in Figure 3.17, the large hole is the location of where the skull
meets the neck. The temporal bone sits at the bottom part of the head.
In Figure 3.19, the author shows an approximate marking for the initial state
Qi and how it would ideally allow the cutting path to move toward Qg. In both
Figures 3.17 and 3.19, the temporal bone is translucent. The opacity of the bone
was reduced to show the location of the critical anatomy, which still obscures the
goal state in that figure. The human in the corner is a reference to show the position
and orientation of temporal bone CT scans. The orientation was specifically chosen
93
Figure 3.18: Model generated of sample No. 3, showing the orientation of the
anatomy during the surgical procedure, with a marker showing the location for the
goal state (Qg = round window).
Figure 3.19: Temporal bone of sample No. 3.
94
to mimic the orientation of a patient as much as possible. Figure 3.18 shows the
location of the goal state Qg, in order to help understand the general path that
needs to be taken.
The rest of the models generated for this dissertation are shown in Appendix
B. This information includes models of the cochlea, and the full temporal bone CT
scans.
3.5 Surface Fitting Equations to Map Out Cochlea
3.5.1 Semicircular Canals
The semicircular canals are the upper part of the cochlea, and these features
are shown in Figure 3.20. The surface fitting generated by MATLAB based on the
segmentation of the CT scans of the semicircular canals from sample No. 3 are
shown below. The equations generated are listed in Eq. (3.3).
f(x, y) = p00 + p10 ? x + p01 ? y + p20 ? x2 + p11 ? x ? y (3.3)
Table 3.2: Polynomial equation coefficients for semicircular canals.
Coeff p00 p10 p01 p20 p11
SCC 25.09 17.22 17.92 -10.36 -0.06441
In Table 3.2, the author shows a fairly good root mean square value ofRMSE =
95
Figure 3.20: Curve fitting generated in MATLAB based on point cloud data of
semicircular canals.
Table 3.3: Statistics on Surface Fitting Equations for Semicircular Canals.
R-square 0.997
Adjusted R-square 0.9997
RMSE 0.1847
? 13.10
mean ? 9.96
0.1847. Second order polynomial equations have been generated from surface fit-
ting in MATLAB with a goodness value of R2 =0.997 (see Figure 3.20). The sur-
face fitting equations have been generated from interpolating some the sample No.3
data points listed in Table 3.1. The calculated population standard deviation is ?
=13.158, with a mean ? = 3.63. While the standard deviation found from this was
was higher than a typically desired value of ? < 1, the standard deviation is simply
stating that the value implies that there is a large spread of data points away from
96
the mean.
3.5.2 Cochlea
Figure 3.21: Polynomial equation coefficients for the cochlea.
The Curve Fitting Toolbox in MATLAB was used to generate a goodness value
of R2 = 0.997 for Figure 3.21. The standard deviation was found to be ? = 12.334,
which is relatively high, but not completely unexpected, given the wide deviation
from the median. The mean turned out to be ? = 2.91512. This value was higher
than a desired value of standard deviation < 1, the standard deviation is simply
stating that there is a large spread of data points away from the mean, which is
expected. Figure 3.21 shows the surface modeling for the lower part of the cochlea.
In Table 3.4, it is demonstrated that the root mean square error is RMSE = 0.3081.
The RMSE was a bit higher than seen in Table 3.2 and also in Table 3.6. Please
97
note that Table 3.6 has the information for the model that will be used in Chapter
4.
Table 3.4: Polynomial equation coefficients for cochlea.
Coeff p00 p10 p01 p20 p11 p02 p30 p21 p12 p03
cochlea 28.7 18.46 -0.6173 -18.75 1.854 -7.391 5.015 0.1721 -0.7876 2.496
Table 3.5: Statistics on Surface Fitting Eqns for cochlea.
R-square 0.997
Adjusted R-square 0.9997
RMSE 0.3081
? 12.334
mean ? 2.91512
3.5.3 Full cochlea model
The cochlea, where the electrode array will be placed, the equations generated
describe the surface fitting.
f(x, y) = p00 + p10 ? x+ p01 ? y + p20 ? x2 + p11 ? x ? y + p02 ? y2
(3.4)
98
Figure 3.22: Curve fitting generated in MATLAB based on point cloud of the full
cochlea for sample No. 3.
Table 3.6: Polynomial equation coefficients for full cochlea.
Coeff p00 p10 p01 p20 p11
Full Cochlea 29.71 27.39 -2.184 -7.268 2.089
In Figure 3.22, the author shows a surface fitting done in the Curve Fitting
Toolbox in MATLAB. Eq. (3.4) that was generated using MATLAB and the Least
Absolute Residuals method. It can be seen that in Table 3.6 that the goodness of fit
measure, R2 = 0.9999, which is a very good fit. Additionally, the RMSE = 0.1907,
and seems to correlate better overall with the work of Biferale et al., (1996) and
Bradshaw et al., (2009), and to a degree also with Pietsch et al., (2018).
When generating the surface fitting equations several types of equations were
99
Table 3.7: Statistics of Surface Fitting equations for full cochlea.
R-square 0.9999
Adjusted R-square 0.9999
RMSE 0.1907
SSE 1036
? 15.49
mean ? 9.9474
considered to model the surface of the cochlea, including polynomial, smoothing
functions, least squares method, cubic functions, and others. After some research,
the author determined that using the curve fitting toolbox in MATLAB would func-
tion better than an independently prepared script, which offered to try to surface
fit the cochlea with a polynomial function, lowess smoothing function, cubic func-
tions, biharmonic functions, nearest neighbors method, and finally a thin plant
spline. After looking at the data with all of these different types, it was found that
a polynomial surface fitting method was the best fit by far. Even when using the
polynomial surface fitting method, orders as low as one (monomial function) as high
as to the fifth power were considered, but ultimately, only going as high as the fourth
power was needed.
3.6 Remarks
Generating these surface equations is found to be beneficial to the PPA, as one
is now able to differentiate between obstacles, and the goal state ahead of time. In
100
addition, as the modeling develops, the PPA will be able to predict how to generate
a path sooner as relationships between critical anatomy are more easily identified.
101
Chapter 4: Cutting Path Planning Algorithms and Updating with
Anatomically Correct Obstacles
In this chapter, the author has separated the path planning algorithms for the
tool path (a.k.a Robot Path), and the Cutting Path. The Cutting Path is used to
generate the path taken from the pinna (outer ear), which is labeled as Qi to the
round window (RW), a natural membrane. This is labeled as Qg.
Identifying the anatomy for the cutting path based on the anatomy would tell
one based on specific metrics of the anatomy if a patient could be operated on by
the surgical system.
The author?s decision to propose the use of a new planning algorithm (pre-
sented in this chapter) is based on several reasons. The author determined that the
negative issues that were present in the RRT method could best be addressed by
a method called Sequential Quadratic Programming (SQP). The issues include the
RRT method being computationally expensive at times, and the response outcome
being a bumpy, node filled, path that is not initially optimized or smooth; see Chap-
ter 2. SQP, on the other hand, is designed to optimize non-linear systems, and to
generate smooth paths, assuming that the objective function can be differentiated
twice.
102
4.1 Literature Review
Constrained optimization Tits, (2018) takes into account equality and inequal-
ity constraints on a system that can have a large impact on the outcome of the overall
result of the system. The Lagrange Multipliers shown in Chapter 2 were used when
only equality constraints were applied to the system. These multipliers can be used
to unconstrain a problem by including a larger number of coordinates than needed
with a constrained problem Tits, (2018).
Navigation and obstacle avoidance for a path planning algorithm require care-
ful thought when dealing with a highly cluttered environment LaValle, (2014). Sieg-
wart et al., (2011), discuss the different methods for navigation in a highly cluttered
environment and how to best deal with various types of obstacles.
When considering how best to approach the cutting path planning environment
and the robot trajectory, it is important to consider the issues that were brought
up in Chapter 2 of this dissertation. A number of PPA models were considered,
but they did not prove to be adequate. Hence, the author chose to also look at an
adapted Sequentially Quadratic Program (SQP) algorithm.
The SQP can be used to solve a non-linear programming (NLP) problem in
an iterative manner by recursively solving a quadratic programming (QP) problem.
At the termination of each iteration, the estimate for the solution is improved by
taking a step in the direction of the solution. SQP requires an initial guess for
each point, and it is important to note that this guess can vastly effect the time to
convergence and even the final solution. As these methods are gradient methods,
103
they are subject to finding locally optimal solutions (e.g. Choset, 2005), which can
be simplified by limiting the program?s working environment. Since this method
can easily take into account a number of constraints while generating the path,
SQP makes for a potentially ideal replacement for RRT, and addressing the issues
that were generated by that modified algorithm.
Figure 4.1: Posterior view mastoid process to see the inner ear, adapted from
CE4RT, (2019).
Figure 4.1 shows the mastoid process and the view of the inner ear from this
posterior view. This view is the view of the surgical path to complete the drilling
process. Carlson, (2018) demonstrates how the beginning of the incision starts
behind the ear, which is also behind the inner ear anatomy, and also at an angle.
Since the cutting path will be starting from a posterior direction, it is good to know
exactly how the anatomy is shaped, so that the algorithm for the drilling path can
recognize the anatomy, no matter what angle the drill path has with respect to parts
of the inner and middle ear. Shown in Figure 4.2 is the drilling path (cutting path)
104
from a different perspective, presented by the authors of the paper by Scorza et al.,
(2021). They discuss the general trajectory for cochlear implantation surgery as
well as other surgical procedures.
Figure 4.2: Generalized drilling path, axial view Scorza et al., (2021).
This desired drilling path is the goal when generating the cutting path with
SQP. Additionally, another goal for this work will be to generate a new robot tra-
jectory to compare with the robot trajectory generated by the RRT algorithm.
4.1.1 Governing Equations for Sequential Quadratic Programming
(SQP) Algorithm
4.1.1.1 Robot Trajectory Governing Equations
The governing equations are equations that are used to determine the different
trajectories in the SQP algorithm. the robot trajectory governing equation is defined
below in Eq. (4.1):
105
Figure 4.3: Desired drilling path to be generated by SQP.
D(qi)q?i + C(qi, q?i)q?i + g(qi) = T (4.1)
As before D is the n? n mass and inertia matrix, C is the matrix of Coriolis
and centripedal forces matrix n ? n, and g is the gravitational forces matrix with
dimensions n? 1. As noted by Lynch and Chongwoo, (2017) centripedal forces can
be interpreted in terms of appropriate velocity components.
Please note that for this governing equation seen in Eq. (4.1), reaction forces
are not considered for the robot arm and end effector along the trajectory for sim-
plification.
106
4.1.1.2 Cutting Path Governing Equations
The governing equation for the cutting path algorithm will be used to generate
the 3D trajectory through the middle ear environment and mastoid bone, which is
shown below:
?
d = ((x ? x )2 + (y ? y )2 + (z ? z )22 1 2 1 2 1 ) (4.2)
where x, y, and z are Cartesian coordinates and d is the distance of the cutting
path.
As shown above, Eq. (4.1) and Eq. (4.2) are the governing equations that will
be used in the SQP algorithms below.
4.2 Task Space Dynamics
While using the joint space dynamics that produces results in terms of the
angles for the robot trajectory presented in Chapter 2, the same method cannot be
applied when dealing with the cutting path planning algorithm. Due to the con-
straints for the cutting path planning algorithm, it is more important to consider the
location of the end-effector from the Cartesian coordinate perspective. Determining
the dynamics of the manipulator arm in the task space is an effective way to resolve
this issue.
In order to accurately map out the end-effector?s position and orientation in
Cartesian space, it is important to consider the dynamics for the task space (e.g.
107
Spong et al., 2012). The task space dynamics are related to the joint space dynamics
(presented in Chapter 2) through Eq. (4.5). In this system, the end-effector forces
are related to the joint torques by the transpose of the Jacobian, which is represented
by J . The torques on the joints is represented by T , and the Cartesian forces and
torques on the end-effector are represented by F.
Recall Eq. (2.1) from Chapter 2. This is equivalent to Eq. (4.3). Note that
the end-effector force is not being considered in this presented work.
T = D(q)q?i + C(qi, q?i)q?i + g(qi) (4.3)
D is the mass and inertia matrix, which is equivalent to D in Eq. (2.1). In
Eq. (4.3), the author considers the centripetal, Coriolis, and gravitational forces
that are dependent on q and its derivatives. It is useful to note that the task space
dynamics are related to the joint space dynamics by Eq. (4.5).
In following the derivations presented in Chapter 8 of the work written by
Spong et al., (2012), one can rearrange Eq. (4.3) to solve for q?i.
Eq. (4.4) is the inverse dynamics equation of motion.
q? ?1i = D(q) (T ? C(qi, q?i)q?i + g(qi)) (4.4)
According to Spong et al. (2012), it is possible to not only go from joint space
to taks space, but also that the following relationship can be determined:
108
X? = J(qi)q?i
X? = J(qi)q?i + J?(q )q? (4.5)i i
where X? is the collected linear acceleration terms in the task space, and J(qi)
is the analytic Jacobian, and due to the 7 DOF system of the KUKA, it is not a
square matrix, dimensions are (p?m). Please note that q are the coordinates of the
system. Finally, Spong et al. (2012) one can remember that the following Jacobian
equation, Eq. (4.6) holds true:
?? ? ? ?
X? = ??v??? ???x?= ??? = J(qi)q?i (4.6)
? ?
where v is the linear velocities in the joint space, and ? is the angular ve-
locities in both the joint space and the task space. This simplifed model for the
relationship between inverse joint space dynamics and task space variables allows
one to determine the position, velocity and acceleration of the end-effector in terms
of cartesian coordinates.
109
4.3 Sequential Quadratic Programming vs. Rapidly-exploring Ran-
domized Trees
4.3.1 Constraints
The robot trajectory (tool path) will use the Constrained Optimization System
that is setup in Chapter 2 of this dissertation.
This could help the surgeon locate a hard to visualize round window, possibly
preventing the need to drill into the cochlear wall. As this program is built up
with a library of statistical data for a given age range and size, it could also help
determine what anatomy in a patient is too fragile for traditional surgical procedural
routes utilized by CIS, and potentially offer alternative drilling paths that can still
allow for the CIS to take place, thereby potentially increasing the feasible set of
patients who receive the treatment for profound hearing loss and deafness. This
map can be refined over time with data for ranges of patients depending on age,
size, abnormality type, etc. In future work, the author hopes to have patients who
were once considered ?in-operable? to become candidates for CIS, with the use of
Path Planning Algorithms.
Non-holonomic constraints tend to be characterized as kinematic constraints,
and these constraints tend to restrict the possible velocities of the system; see,
for example, Greenwood, (1997). Both holonomic and non-holonomic constraint
equations are used for the systems tested and simulated in Chapters 2 and 4 of
this dissertation. From Greenwood, (1997), the non-holonomic constraints can be
110
written as:
?n
ajidqi + ajtdt = 0 (4.7)
i=1
where j = 1, 2, ..., s, and t is time. Holonomic conditions can be used to reduce
the number of coordinates needed, while with non-holonomic constraints, one needs
more coordinates than the number of degrees of freedom. These equations shown
below can be used to check holonomic constraints.
?aji ?ajk
=
?qk ?qi (4.8)
?aji ?ajt
=
?t ?qi
where i, k = 1, 2, ..., r. a are considered functions of the coordinates qi and t.
The dimensions used to generate boundary conditions below will be evaluated with
the above condition. Although a drilling path would ideally be a straight-line, the
proposed drilling path in three dimensions needs to be determined. To that end, the
author uses an objective function with an initial state and goal state for determining
the course of the path.
4.3.2 Constraints for the Cutting Path
As noted in Chapter 3, McRackan et al. (2012) found that the pediatric
middle and inner geometric relationships are substantially different from those of a
respective adult inner and middle ear. Therefore it is important to clearly note the
geometry obtained from the CT scans for this dissertation. The general equation
111
for a line in 3D is represented as shown in the following manner shown in Eq. (4.9).
Please note that the data from Sample 3 is used for this.
The following equation is a constraint applied to the Cutting Path, represent-
ing the nearest line edge of the cylindrical drill path to the Malleus, Incus, Stapes,
and Cranial Nerve (CN7). As opposed to the robot trajectory PPA, this system will
optimize the length of the cutting path, while respecting the constraints.
Li = Qg + t(Qg ?Qi) t ? R
(4.9)
Here, the initial state and goal state are represented as follows:
?? ? ? ???x? g???? ???
x
? i?? ?? ?? ?
??
Qg = ?y ?Qi =g ?yi???? (4.10)
zg zi
The cutting path can be represented in parametric form as the following equa-
tions:
Lx = xg + t(xg ? xi)
Ly = yg + t(yg ? y ) (4.11)i
Lz = zg + t(zg ? zi)
These boundary conditions derived from the dimensions below will be used as
inequality conditions for the system.
112
Table 4.1: Dimensions used to generate boundary conditions for the cutting path
(Sample 3 Data).
Quantities Dimensions for boundary conditions
Diameter of RW 2.53 mm
Thickness of RW 0.83 mm
Length of Cochlea 15.2 mm
Width of Cochlea (vest.) 7.639 mm
Diameter of BT 1.88 mm
diameter of CN7 1.48 mm
length of CN7 16.03 mm
Distance from BT to Apex 5.73 mm
TB Length 98.17 mm
TB Width 76.05 mm
TB Thickness 35.68 mm
4.3.3 Constraints for the tool path
Both the robot trajectory (tool path) and the Cutting Path use the same
objective function listed in Chapter 2, as there is no distance along the path to
minimize. The major difference is the constraints that are applied to the system, and
what the obstacle avoidance algorithm dictates. Most of the boundary constraints
are geometric in nature, as the equations are describing a highly cluttered and tight
working environment.
113
subject to:
?
? ??vi 0.05m/s ????????
ai ? 0.10m/s2?????
??
??
??i < 90 deg/s
??
???????
??i < 25 deg/s
s ???????
0 ? ?1 ? 170o ???????
?
??
0 ? ??2 ? 120o ???? for Constrained Optimization of Robot Path
0 ? ? ? 170o ?3 ?????????????0 ? ?4 ? 120o
0 ? ?5 ? 170o ??????
??????
0 ? ? ? 120o6 ???????
???
dmax ? ?.8m ??????????
(4.12)
All the constraints shown in Eq. (4.12) are considered non holonomic as they
are not integrable. These restrictions on the KUKA LWR are the limitations set
out by the company that creates the KUKA LWR, so therefore these restrictions
would need to be taken into account for the system.
Sequential Quadratic Programming method is a subset of nonlinear optimiza-
tion. The objective function is converted as shown below. The path from the initial
114
state to the goal state can be represented as a line in three dimensions, as is repre-
sented by governing equation for the cutting path, Eq. (4.2).
Next, it is important to define the distance from the cutting path to the nearest
point on the bounding boxes surrounding the delicate anatomy that is to be avoided.
The equation below will simultaneously help determine the location of the cutting
path from the nearest point of the boundary buffer.
xi = Xpath + ?(xspt) (4.13)
yi = Ypath + ?(yspt) (4.14)
zi = Zpath + ?(zspt) (4.15)
(x + n)2 + (y + n)2 + (z + n)2i i i (4.16)
where n is a buffer distance in mm, set for the system, and ?(xspt, yspt, zspt)
is a proportional value that is in the direction of the vector of the cutting path.
The boundary condition shown in Eq. (4.16) would allow for the application of a
bounding box around the malleus, incus, and stapes, and cranial facial nerve. This is
intended to help protect the highly sensitive anatomy, while still satisfying the need
to properly constrain the system. Additionally, Eq. (4.16) represents the constraint
that tells the program that the cutting path must be distance 1.2 mm from the
obstacle regions, according to the work presented by McRackan, (2012), in order
to better assure protection. This equation also builds in a buffer for the critical
anatomy. This dual system to create a buffer between the drill path is especially
115
important since Carlson, (2018) demonstrated how the cutting path for a pediatric
case may have multiple twists and turns. Since SQP recalculates on an iterative
basis, this system is more likely to keep determining the cutting path?s projected
distance from the multiple boundary constraints.
Since the system is a nonlinear constrained system, it is best to look at a
method that will allow for optimization under difficult circumstances. The author
determined that the method Sequential Quadratic Programming would be a good
candidate for evaluating such a heavily constrained system, as the approach is iter-
ative, and automatically takes into account the different constraints on the system.
This method starts out with the method set out in Chapter 2, for dealing with
constrained optimization problems.
To determine how the dynamics are affected, the equation of motion demon-
strated given by Eq. (2.1), is adapted to the following constrained Lagrangian, for
testing in what is known as the Karush-Kuhn-Tucker (KKT) conditions Nash and
Sofer, (1996). KKT is a first order derivative used to determine an optimal solution
in a nonlinear optimization problem Nash and Sofer, (1996).
The Lagrangian of this constrained problem is generated below, and the op-
timality conditions are determined to be satisfied, as shown below. Determining
whether the optimality conditions are satisfied will be the basis for writing the path
planning algorithm shown in this presented work Bertsekas, (1996).
The KKT equations were evaluated,with the following equation:
116
?? ???g(x)? A(x)?T???L(x, ?) ?? = 0 (4.17)
c(x)
Recall that ?L(x, ?) are the optimality conditions determined by the KKT
test. c(x) is holonomic constraint applied to the system, and the matrix A is an
n?m matrix, and is generally non symmetric. g(x) ? ?f(x), and is an n vector.
Next, the Hessian of the Lagrangian for the constrained system is determined.
The Hessian helps determine the maximum or the minimum of a multi-variable
system Bertsekas, (1996). The Hessian, W is denoted as:
W (xk, ?k) = ?2xxL(x, ?) (4.18)
The Hessian, which is the second partial derivative matrix, is then used to
determine the Lagrange multiplier steps pk, p? via the Newton Method, shown below
in Eq. (4.19).
?? ?? ? ? ???W(xk, ?k A?T ?Tk ?k??????pk??? ????gk + A ?k?= ?? (4.19)
Ak 0 p? ?ck
Where, c The Newton step is given by:
?? ?? ? ? ? ???xk+1?? = ???xk???+ ???pk??? (4.20)
?k+1 ?k p?
Once the Lagrange multiplier steps have been determined, it is possible to set
117
up the pseudocode for the system.
4.3.4 Pseudocode For Sequential Quadratic Programming
Algorithm 1: Algorithm using sequential quadratic programming (SQP)
to determine a path from Qi to Qg.
1 Input: Initial guess(x 0,? 0 parameters O ? ? ? 0.05 Output: Optimum, x?
k ? 0 Initialize the Hessian , W0 ? I repeat
2 o
3 until C ;
4 mpute Pk and P? by solving using Newtons Method, with Wk while B do
5 o
6 undary Constraints are not violateddo Determine xk+1 and ?k+1 Evaluate
Hessian Wk+1 terms until system converges
The intent through producing the pseudocode is to show the overall flow of
the path planning algorithms for the robot trajectory and the Cutting Path. The
idea behind utilizing this is to simultaneously consider the boundary restrictions
generated for the critical anatomy as well as minimize the amount of torque input
into the system. This allows for a safer and more controlled process of completing
the cutting path.
4.4 Results
4.4.1 Results for Robot Trajectory
In the Figures 4.4, 4.5, and 4.6, one can note that the robot trajectory was
able to find an optimal solution for a constrained system. The figures show the
results for the robot trajectory as it follows a path determined by SQP for the robot
118
trajectory. It is easy to note how smooth the path is for each figure. While the path
does not follow a perfectly straight line, it represents a marked improvement over
the previous algorithm, RRT.
1. The red points are the times that the robot trajectory came close to violating
the constraints.
2. ?1 = black path
3. ?2 = dark blue path
4. ?3 = cyan path
5. ?4 = green path
6. ?5 = yellow path
7. ?6 = gray path
8. ?7 = pink path
Figure 4.4: Angular position (? i) of the KUKA LWR in the SQP optimized system.
For the robot trajectory, open loop dynamics were used to simplify the matter,
as the KUKA system is a 7 DOF system. In the future, in order to pursue closed
loop avoidance, haptic feedback sensors as well as vision sensors may be needed.
119
1. The red points are the times that the robot trajectory came close to violating
the constraints.
2. ?1 = black path
3. ?2 = dark blue path
4. ?3 = cyan path
5. ?4 = green path
6. ?5 = yellow path
7. ?6 = gray path
8. ?7 = pink path
Figure 4.5: Angular speed ?? i of the KUKA LWR in the SQP optimized system.
120
1. The red points are the times that the robot trajectory came close to violating
the constraints.
2. ?1 = black path
3. ?2 = dark blue path
4. ?3 = cyan path
5. ?4 = green path
6. ?5 = yellow path
7. ?6 = gray path
8. ?7 = pink path
Figure 4.6: Minimized torque in the SQP optimized system.
121
4.5 Remarks
In evaluating how effective the robot trajectory path generated by the PPA is,
one has to consider the following: (1) smoothness of the final path, (2) the frequency
with which the robot trajectory has to change direction, and finally, (3) the amount
of torque needed to control the system. There was only a minimal amount of torque
needed, as seen in Figure 4.6.
Due to the relatively small sample size presented in this dissertation, there
were no statistically significant trends for comparing the drilling trajectory to cur-
rent methods of completing CIS. A next step for this work would be to conduct
experimental testing and compare to current methods for CIS.
As the program is iterated, more and more realistic constraints should be
added to the cutting path. A world frame for the robot trajectory and for the
cutting path will be considered.
122
Chapter 5: Concluding Remarks
A summary of contributions made through this dissertation work and plans
for future work are outlined here.
5.1 Summary of Contributions
This is the first time that simulated studies and a 3D path planning algorithm
(PPA) have been specifically developed for pediatric cochlear implantation surgery.
By studying the constraints and system dynamics for the tool path, the author
has strived to add an additional layer of safety to the surgical procedure associated
with a cochlear implant. Typically, kinematics has been used to control robotics
systems when used in surgical procedures due to the very low speeds, and limited
accelerations. While the full equations of motion have not typically been used in
PPA for surgical robotics, or most surgical applications; however, the ability to
control the input torque on the system adds for an additional layer of safety that is
desirable in a tight and highly cluttered surgical environment such as in a temporal
bone.
The PPA presented in this dissertation, the author has created the founda-
tional elements necessary for a cutting PPA and a Robot Trajectory (tool path)
123
that would the associated robot to assist the surgeon in the operating theater and
mitigate issues when working in a tight, highly cluttered anatomical environment,
and to potentially reduce mastoid bone loss, and potentially reduce operation time.
5.2 Recommendations for Future Work
The author recommends continued studies with this PPA and the progression
towards experimental testing so that the PPA can be compared to manual surgical
drilling for quantitative analysis and compared more quantitatively to current CIS
procedures, taking into account surgical plans to avoid other possible issues such
as thermal injury during drilling. Additionally, a future goal of such work can be
to recognize types of malformations in the middle ear anatomy as well as pertinent
surrounding structures and tissues. Plastic model experimental tests should be con-
ducted to determine if the simulation times for completion of drilling are accurate,
and if the promoted reduction in bone loss would be validated.
With regard to experimental testing, the following points are made: A 3D
model of a temporal bone would need to be printed in plastic, and marked with
fiducials (a marker typically used with imaging systems, used as a point of reference).
This would then be progressively tested with a 3D model of a complete skull. This
is where the MATLAB Bridge to Slicer would have further use, since the locations
of the ficudials can be implemented in Slicer to aid the semgementation process.
These fiducial markers, when used in coordination with an external reference frame,
can subsequently be used to orient the PPA as markers in both Slicer and on the 3D
124
plastic models. In order to do this, the program would need to be adapted to allow
for visual feedback from an external source, such as Microsoft Kinect or a similar
device. The fiducials need to be of standard size and recognizable by the PPA as
separate from obstacles and from an an initial state or a goal state.
A separate PPA could be written for the insertion of the cochlear implant
electrode array, and potentially added to complete the entire procedure.
Stringent control over the machinery is needed to ensure absolute obstacle
avoidance when dealing with a millimeter scale environment. In this case, the torque
generated by utilizing the dynamics is beneficial. The different constraints applied
to the robot trajectory (tool path) PPA can be in terms of speed, applied torque,
restriction of angular rotation.
The separate PPA has been modified in an iterative manner, as discussed in
this dissertation work. The cutting PPA can be generated separately from the tool
PPA. A 3D PPA that makes use of governing physics models for mapping out a
known, but still complex and tightly compacted environment with multiple delicate
structures and obstacles would be desirable. The author?s algorithm expands the
field by including specific constraints and obstacle avoidance to preserve critical
anatomy. This makes the PPA more accurate and is expected to generate realistic
paths for patients with abnormally shaped inner ears or congenital defects.
The author has used 5 anatomically accurate 3D models in the cutting PPA
that has been updated to include non-holonomic constraints and the obstacle avoid-
ance algorithms are being adapted to make better use of anatomically accurate
models and differentiate obstacles and the goal state. The introduction of the geo-
125
metric constraints can allow one to ensure that the path generated is more realistic.
5.3 Additional Remarks
When considering the design of a PPA that would generate the path for the
electrode array to enter the cochlea, it is important to consider how the robot can
be used to perform the surgery, while minimizing the trauma at the implantation
site, as well as preventing accidental nicks, tears, and assuring the required drilling
depth. Examining the different styles of cochlear implantation will determine which
method is better suited for a robotic surgery.
126
Appendix A: Medical Terminology and Images
In this medical terminology section, the medical terms that relate to the outer,
middle, and inner ear are presented for reference in this dissertation. It is pertinent
to know the different medical terminology for the path planning algorithms.
Figure A.1: External Auditory Canal, adapted from Trust (2019.)
The Outer Ear
Pinna ? outer ear and is made of rigid cartilage, and covered by skin (Hoffman,
2019).
127
Figure A.2: Reference Planes and Frames, adapted from College (2019).
128
Figure A.3: Body-centric reference frames- coronal, sagittal, and frontal, adapted
from Terray (2016).
Figure A.4: Coronal Section View of the Mastoid Air Cells, adapted from Gray
(2019).
129
Figure A.5: Interior Portions of the cochlea, adapted from Park (2019).
External Auditory Canal (EAC) ? short tubes that end at the eardrum (tym-
panic membrane) (Alberti, 2006; Hoffman 2019).
The Middle Ear
Auditory Ossicles: These are the smallest bones in the body, they connect the
tympanic membrane to the oval window in the vestibule (McGovern, 2008). They
are located in the tympanic cavity (Hawkins, 2019).
1. Malleus ? (hammer), its handle is embedded in the tympanic membrane,
running from its center upwards. The head of the club lies in a cavity of the middle
ear above the tympanic membrane (Alberti, 2006; Hawkins, 2019).
2. Incus ? (anvil) runs backwards from the malleus and has sticking down
from it a very little thin projection known as its long process which hangs freely in
130
the middle ear. It has a right angle bend at its tip (Alberti, 2006; Hoffman, 2019).
3. Stapes ? (stirrup) The smallest bone, which covers the oval window that
sits in the vestibule. Sound is transmitted directly to the oval window.
Eustachian Tube ? (auditory tube, Internal Auditory Meatus, Internal Audi-
tory Canal) is a tube that links the nasopharynx to the middle ear, it is part of the
middle ear ( Hoffman, 2019; Hawkins, 2019). At the front end of the middle ear lies
the opening of the Eustachian tube, and at its posterior end is a passageway to a
group of air cells within the temporal bone known as the mastoid air cells. (Alberti,
2006; Hawkins, 2019). The facial and vestibulocochlear nerves pass through the
internal acoustic meatus, alongside the vestibular ganglion and labyrinthine artery.
Mastoid Antrum ? The antrum is a large air cell superior and posterior to the
tympanic cavity and connected to the tympanic cavity via aditus ad antrum (Beek
and Pameijer, 2017).
Perilymph ? fluid in Scala Tympani and Scala Vestibuli (Hoffman, 2019).
Tympanic Membrane ? also called eardrum, is the boundary between the outer
ear and the inner ear. It is made of thin tissue in the human ear that transmits
sound to the auditory ossicles, which are tiny bones in the tympanic (middle-ear)
cavity. The membrane stretches across the end of the external canal and resembles
a thin cone with its tip. (Beek and Pameijer, 2017; Alberti, 2006; Hawkins, 2019).
When sound hits the tympanic membrane, it creates a wave in the fluid of the inner
ear, which passes through the cochlea. An electrical impulse is sent to the 8th cranial
nerve (McGovern, 2008).
tympanic cavity ? connects to the nasopharynx anteriorly and medially by the
131
pharyngotympanic tube and posteriorly and superiorly by the mastoid antrum, a
cavity in the mastoid process of the temporal bone, by the aditus to the mastoid
antrum (Hawkins, 2019). The tympanic cavity is an air chamber; it contains a chain
of movable bones which transmit the vibrations of the tympanic membrane across
the cavity to the middle ear.
The Inner Ear
cochlea ? The part of the inner ear responsible for hearing. It takes the sound
waves and processes them so that they can be transmitted to the brain via the
vestibulocochlear nerve. This is the ?shell? part connected to the Vestibule and the
Semicircular Canals.
endolymph ? fluid in the Semicircular Canals (Costa, 2019) and (Hawkins,
2019).
organ of corti ? in one of the three compartments of the cochlea, has the hairs,
that allow for hearing (Nave, 2017).
oval window ? (stapes footplate sits on the the Oval Window). The oval
window is a flexible membrane that transfers vibrations into the cochlea, causes a
wave in the perilymph fluid (McGovern, 2008).
round window? Flexible membrane along lower region of cochlea. This mem-
brane is also responsible for hearing, and is often used for CIS (McGovern, 2008).
Works in a reciprocal manner from the oval window. The Round Window bulges out
when the Perilymph carrying sound waves reaches it, which then causes an electrical
impulse to be sent along the auditory nerve to the brain (Sahyouni, 2014).
scala tympani ? a lower chamber of the cochlea (McGovern, 2008), on the
132
inside.
scala vestibuli ? an upper chamber of the cochlea (McGovern, 2008), on the
inside of the cochlea.
vestibule ? partially responsible for balance, connects the Semicircular Canals
and the cochlea.
semicircular canals ? The three semicircular canals of the bony labyrinth are
designated according to their position: superior, horizontal, and posterior. Partially
responsible for balance.
Nearby Anatomy
cochlear Nerve ? (Auditory Nerve, part of Cranial Nerve # 8) The cochlear
nerve carries auditory sensory information from the cochlea of the inner ear directly
to the brain.
cranial facial nerve ? (cranial nerve # 7) responsible for controlling muscles in
the face of the patient.
vestibular nerve ? (part of Cranial Nerve # 8) attached to the semicircular
canals. The vestibular nerve transmits afferent signals from the labyrinths through
the internal auditory meatus. Connects to the vestibulocochlear nerve.
vestibulocochlear Nerve (Cranial Nerve # 8)? connects the cochlear nerve and
the vestibular nerve.
mastoid antrum ? is a large air cell superior and posterior to the tympanic
cavity and connected to the tympanic cavity via the aditus ad antrum (Beek and
Pameijer, 2017).
temporal bone ? contributes to the lower lateral walls of the skull. It contains
133
the middle and inner portions of the ear, and is crossed by the majority of the cranial
nerves (Beek and Pameijer, 2017). Mastoid Part of the Temporal Bone ? Mastoid
air cells, large and small air cells.
Clinical Terms
Cochleostomy- Opening of the inner ear for inserting the electrode. In CIS,
Cochleostomy refers to when a hole is drilled into the cochlea in order to insert the
electrode array of the cochlear implant (Hawkins, 2019)
Mastoidectomy - the surgical process to remove the mastoid air cells (Beek
and Pameijer, 2017).
Directions
Anterior ? In front of the body (Costa, 2019).
Cranial ? Head, towards the head (Costa, 2019).
Caudal ? Tail, towards the tail (Costa, 2019).
Dorsal ? ?towards the back?, upper side, interchangable with posterior (Klieg-
man, 2007).
Distal ? farthest away from the trunk of the body relative to another anatom-
ical part, can be used to describe regions of the body (Costa, 2019).
External ? outside of the body, exposed to the outside environment (Costa,
2019).
Inferior ? Lower or below, towards the feet (Costa, 2019).
Internal ? inside of the body, not exposed to the outside environment (Costa,
2019).
Lateral ? This is the opposite of Medial. This term is used to describe the
134
relative distance of a part or region of the body from the Median.
Medial ? this term is used to describe distance from the Median (relative
closeness of body part to Median).
Median ? An invisible line that splits the body into two equal halves down the
mid-line of the body, follows the Sagittal Plane. Not to be confused with Medial
(Costa, 2019).
Posterior ? Behind the body (Costa, 2019).
Proximal ? Describes something that it close to the trunk of the body (Costa,
2019). This can be used to describe regions of the body, or specific section of a body
part.
Rostral ? towards the head or face (Costa, 2019).
Superior ? Upper or above, towards the top of the head (Costa, 2019).
Ventral ? ?towards the abdomen?, synonymous with Anterior (Kliegman,
2007).
Reference Frames
Coronal Plane ? (Frontal Plane) A vertical plane that divides the body into
Anterior and Posterior.
Sagittal Plane ? (Lateral Plane- Median Plane, Mid-Sagittal Plane) a vertical
divides the body into left and right. Median (Midline) runs along this plane (Costa,
2019).
Transverse Plane ? (Axial Plane) a horizontal plane that divides the body into
Superior and Inferior sections (Costa, 2019).
135
Appendix B: Additional Data
Figure B.1: The polynomial fit for the segmented models of the cochlea generated
in MATLAB.
B.1 Figures showing measured dimensions
The following sample of dimensions illustrates the segmented models that were
generated in Slicer, and then were subsequently measured in solidworks.
B.2 MATLAB code for generating surface equations
Please note that this code is for the full cochlea model illustrated in Figure
3.16, and can be seen in Chapter 3.
136
Figure B.2: Segmented Model of Sample No. 2. cochlea and round window.
137
Figure B.3: Segmented Model of Case No 4. Cochlea and Round Window,
Semicircular Canals show more mild malformations than in Case No 1.
138
Figure B.4: Sample No. 1 cochlea and round window segmentation.
Figure B.5: Sample No. 2 cochlea and round window segmentation.
139
Figure B.6: Sample No. 4 cochlea and round window segmentation.
140
Figure B.7: Sample No. 5 cochlea and round window segmentation.
Figure B.8: The polynomial fit for the segmented models of the semicircular canals
generated in MATLAB.
141
Figure B.9: Length of cranial facial nerve (CN7).
Figure B.10: Approximate diameter of cranial facial nerve (CN7).
142
Figure B.11: Diameter of the round window membrane (RW).
Figure B.12: Approximate length of the temporal bone from Sample No. 3.
143
Figure B.13: Width of cochlea.
Figure B.14: Length of cochlea.
144
 
 
 
 
References 
 
 
L. Adhami and E. Coste-Maniere. Optimal planning for minimally invasive surgical 
robots. IEEE Transactions on Robotics and Automation, pages 854?863, 2003. 
10.1109/TRA.2003.817061. 
P.  Aebischer,  G.  Mantokoudis,  S.  Weder,  L.  Anschuetz  M.  Caversaccio,  and 
W. Wimmer. In-vitro study of speed and alignment angle in cochlear implant 
electrode array insertions. IEEE Transactions On Biomedical Engineering, pages 
129?137, January 2022. URL https://pubmed.ncbi.nlm.nih.gov/34110987/. 
N. Aghakhani, M. Geravand, N. Shahriari, M.  Vendittelli,  and  G.  Oriolo.  Task 
control with remote center of motion constraint for minimally invasive  robotic 
surgery. In 2013 IEEE International Conference on Robotics and Automation 
(ICRA), pages 5807?5812, 05 2013. ISBN 978-1-4673-5641-1. 10.1109/ICRA 
.2013.6631412. 
P. Alberti. The anatomy and physiology of the ear and hearing. In The Encyclopae- 
dia Brittanica, pages 1?36, 2006. 
H. Asada and J.-J. Slotine. Robot analysis and control. J. Wiley and Sons, 1986. 
ASHA. Cochlear implants. https://www.asha.org/public/hearing/Cochlear 
-Implant/, 2019. 
A.B. Auinger, V. Dahm, R. Liepins, D. Riss, W.-D. Baumgartner, and C. Arnolder. 
Robotic cochlear implant surgery: Imaging-based evaluation of feasibility in clin- 
ical routine. Frontiers in Surgery, pages 1?8, September 2021. URL https:// 
www.frontiersin.org/articles/10.3389/fsurg.2021.742219/full. 
D.D. Backous. Cochlear implant placement: round window approach. Neurosurgical 
Focus, 36(v1supplement):1, 2014. 10.3171/2014.v1.focus13537. URL http:// 
thejns.org/doi/abs/10.3171/2014.V1.FOCUS13537. 
E. Beek and F. Pameijer. Temporal bone - pathology. http:// 
www.radiologyassistant.nl/en/p49c62abe0880e/temporal-bone-pathology 
.html#i49c62b7f14f0c., 2017. 
D. P. Bertsekas. Constrained optimization and Lagrange multiplier methods. Athena 
Scientific, 1996. 
 145 
L. Biferale, R. Benzi, R.M. Kerr, and E. Trovatore. Helical shell models for three 
dimensional turbulence. Advances in Turbulence VI Fluid Mechanics and its Ap- 
plications, page 197?200, 1996. 10.1007/978-94-009-0297-8 56. 
R.  Bischoff,  J.  Kurth,  G.  Schreiber,  R.  Koeppe,  A.  Albu-Scha?effer,  A.  Beyer, 
O. Eiberger, S. Haddadin, A. Stemmer, G. Grunwald, and G. Hirzinger. The 
kuka-dlr lightweight robot arm - a new reference platform for robotics research 
and manufacturing. In ISR / ROBOTIK 2010, volume 2, pages 1?8. KUKA 
Automation Company, 01 2010. 
P.T. Boggs and J.W. Tolle. Sequential quadratic programming*. Acta Numerica, 
pages 1?100, 1996. URL .Availablefrom:http://web.cse.ohio-state.edu/ 
~parent.1/classes/788/Au10/OptimizationPapers/SQP/actaSqp.pdf. 
A.P. Bradshaw, I.S. Curthoys, M.J. Todd,  J.S. Magnussen,  D.S. Taubman,  S.T. 
Aw, and G.M. Halmagyi. A mathematical model of human semicircular canal 
geometry: A new basis for interpreting vestibular physiology. Journal of the 
Association for Research in Otolaryngology, 11(2):145?159, 2009. 10.1007/s10162 
-009-0195-6. 
M. Carlson. https://www.youtube.com/watch?v=y1vYjXjL9bs, Mar 2018. 
M. Caversaccio,  K. Gavaghan,  W. Wimmer,  T. Williamson,  J. Anso?,  G. Mantok- 
oudis, N. Gerber, C. Rathgeb, A. Feldmann, F. Wagner, O. Scheidegger, M. Kom- 
pis, C. Weisstanner, M. Zoka Assadi, K. Roesler, L. Anschuetz, M. Huth, and 
S. Weber. Robotic cochlear implantation: surgical procedure and first clinical 
experience.  Acta oto-laryngologica, 137:1?11, 02 2017.  10.1080/00016489.2017 
.1278573. 
CE4RT. X-ray  positioning  of  the  mastoid  process  for  radiologic  techs. 
https://ce4rt.com/positioning/radiographic-positioning-of-the 
-mastoid-process/, Apr 2019. 
Y. Chen, X. Zhao, and J. Han. Review of 3d path planning methods for mobile 
robot. Robot, 32(4):568?576, 2010. 10.3724/sp.j.1218.2010.00568. 
T. Chin. Robotic path planning: Prm & prm*. Medium, pages 1?11, Feb 
2019. URL https://theclassytim.medium.com/robotic-path-planning-prm 
-prm-b4c64b1f5acb. 
H.M. Choset. Principles of robot motion: theory, algorithms, and implementation. 
MIT Press, 2005. 
Northern Virginia Community College. Biology ii. https://courses 
.lumenlearning.com/biology2xmaster/chapter/features-used-to 
-classify-animals/, 2019. 
KUKA Automation Company. Spez lbr iiwa en, lbr iiwa 7 r800 en, lbr iiwa 14 r820 
en. Technical report, Augsburg,Germany, 03 2019. 
 146 
J. Costa. Directional terms and body planes. https://www.udemy.com/human 
-anatomy-basics/learn/lecture/5669132#overview, May 2019. 
H. Eilers, S. Baron, T. Ortmaier, B. Heimann, C. Baier, T. Rau, M. Leinung, and 
O. Majdani. Navigated, robot assisted drilling of a minimally invasive cochlear 
access. In IEEE ICM 2009, pages 1 ? 6, 05 2009. 10.1109/ICMECH.2009.4957213. 
A.   Gasparetto,    P.   Boscariol,    A.   Lanzutti,    and   R.   Vidoni.      Path   plan- 
ning   and   trajectory   planning   algorithms:      A   general   overview.        Mo- 
tion and Operation Planning of Robotic Systems Mechanisms and  Ma- 
chine   Science,   page   3?27,   2015.       10.1007/978-3-319-14705-5 1.       URL 
.Availablefrom:https://www.researchgate.net/publication/282955967  Path 
Planning and Trajectory Planning Algorithms A General Overview. 
C. Gaz, F. Flacco, and A. De Luca. Identifying the dynamic model used by the 
kuka lwr: A reverse engineering approach. 2014  IEEE  International  Conference 
on Robotics and Automation (ICRA), pages 1386?1392, 2014. 
M. Gray. The middle ear. https://teachmeanatomy.info/head/organs/ear/ 
middle-ear/, 2019. 
P. Gray. Vestibule, Jan 2022. URL https://www.open.edu/openlearn/science 
-maths-technology/biology/hearing/content-section-3.1. 
D.T. Greenwood. Principles of Dynamics, chap. 5. Dover Publications, 1997. 
A.X. Hang, G.G. Kim, and C.J. Zdanski. Cochlear implantation in unique pediatric 
patients. Curr. Opin. Otolaryngol Head Neck Surg, pages 1?17, December 2012. 
URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3512207/. 
J.E. Hawkins. Human ear, 2019. URL https://www.britannica.com/science/ 
ear. 
M. Hoffman. Picture of the ear: Ear conditions and treatments, 
2019. URL https://www.webmd.com/cold-and-flu/ear-infection/picture 
-of-the-ear. 
Johns Hopkins Health System. Cochlear implant surgery. https:// 
www.hopkinsmedicine.org/health/treatment-tests-and-therapies/ cochlear-
implant-surgery, 2019. 
C.T.  John. Slicer?s  coordinate  systems. www.slicer.org/wiki/Coordinate 
systems, May 2019. 
S.Y. Ko and F. Rodriguez. Toward a miniaturized needle steering system with path 
planning for obstacle avoidance. IEEE transactions on bio-medical engineering, 
60, 11 2012. 10.1109/TBME.2012.2227741. 
 147 
J. Kronenberg, L. Migirov, and T. Dagan. Suprameatal approach:  new  surgical 
approach for cochlear implantation. The Journal of Laryngology & Otology, 115 
(04), 2001. 10.1258/0022215011907451. 
R.F. Labadie, R. Balachandran, J.H. Noble, G.S. Blachon, J.E. Mitchell, F.A. Reda, 
B. M. Dawant, and J.M. Fitzpatrick. Minimally invasive image-guided cochlear 
implantation surgery: First report of clinical implementation. The Laryngoscope, 
124(8):1915?1922, 2014. 10.1002/lary.24520. 
S.M. LaValle. Rapidly-exploring random trees: A new tool for path planning. Tech- 
nical report, Iowa State University, 1998. 
S.M. LaValle. Planning algorithms. Cambridge University Press, 2014. 
S.M. LaValle and J.J. Kuffner. Randomized kinodynamic planning. International 
Journal of Robotics Research, 20(5):378?400, 5 2001. ISSN 0278-3649. 10.1177/ 
02783640122067453. 
K. Lynch and F. Chongwoo. Modern Robotics. Cambridge University Press, 2017. 
O. Majdani, T.S. Rau, S. Baron, H. Eilers, C. Baier, B. Heimann, T. Ortmaier, 
S. Bartling, T. Lenarz, and M. Leinung. A robot-guided minimally invasive 
approach for cochlear implant surgery: preliminary results of a temporal bone 
study. International Journal of Computer Assisted Radiology and Surgery, 4(5): 
475?486, 2009.   10.1007/s11548-009-0360-8.   URL https://www.ncbi.nlm.nih 
.gov/pubmed/20033531. 
McGovern. Uthealth?otorhinolaryngology ? head & neck surgery. https://med 
.uth.edu/orl/online-ear-disease-photo-book/chapter-3-ear-anatomy/ 
ear-anatomy-inner-ear/, 2008. 
E. McPhillips. World wide hearing loss: Stats from around the world. https://www 
.audicus.com/world-wide-hearing-loss-stats-from-around-the-world/, 
Mar 2019. 
T.R McRackan, F.A.  Reda,  A.  Rivas,  J.H.  Noble,  M.S.  Dietrich,  D.M.  Dawant, 
and R.F.  Labadie.  Comparison  of  cochlear  implant  relevant  anatomy  in  chil- 
dren versus adults. Otology & Neurotology, 33(3):328?334, 2012. 10.1097/ 
mao.0b013e318245cc9f. 
C.A. Megerian. Cochlear implant surgery treatment and management: Medical 
therapy, surgical therapy, preoperative details. https://emedicine.medscape 
.com/article/857242-treatment#d14, Apr 2019. 
F. Nageotte, P. Zanne, M.D. Mathelin, and C. Doignon. A circular needle path 
planning method for suturing in laparoscopic surgery. Proceedings of the 2005 
IEEE International Conference on Robotics and Automation, page 514?519, Apr 
2005. 10.1109/robot.2005.1570170. 
 148 
S.G. Nash and A. Sofer. Linear and Nonlinear Programming. MCGRAW HILL 
SERIES IN INDUSTRIAL ENGINEERING AND MANAGEMENT SCIENCE. 
McGrawHill, 1996. 
C.R. Nave. Organ of corti. http://hyperphysics.phy-astr.gsu.edu/hbase/ 
Sound/corti.html, 2017. 
NIH. Cochlear implants. https://www.nidcd.nih.gov/health/cochlear 
-implants, Jun 2018. 
NIDCD Office of Health Communication and Public Liaison.  Cochlear implants. 
https://www.nidcd.nih.gov/health/cochlear-implants, Jun 2018. 
K. Panara, D. Shahal, R. Mittal, and A.A. Eshraghi. Robotics for cochlear implan- 
tation surgery: Challenges and opportunities. Otology & Neurotology, Inc., pages 
e825?e835, August 2021. URL https://pubmed.ncbi.nlm.nih.gov/33993143/. 
R. Park. inside cochlea. https://www.purposegames.com/game/inside-cochlea, 
2019. 
M. Pietsch, L. Da?vila, P. Erfurt, E. Avci, T. Lenarz, and A. Kral. Publisher correc- 
tion: Spiral form of the human cochlea results from spatial constraints. Scientific 
Reports, 8(1), 2018. 10.1038/s41598-018-25325-8. 
J. Pile, C. M. Yi, J. Zhang, and N. Simaan.  Algorithms and design considerations 
for robot assisted insertion of perimodiolar electrode arrays. In 2011 IEEE In- 
ternational Conference on Robotics and Automation, pages 2898?2904, 05 2011. 
10.1109/ICRA.2011.5979987. 
C. Poley and B. Balachandran. Motion analysis of robot arm with movement re- 
striction. In ASME 2016 International Mechanical Engineering Congress and 
Exposition, volume Volume 4A: Dynamics,  Vibration,  and Control,  pages 1? 
9, 11 2016. 10.1115/IMECE2016-65513. URL http://dx.doi.org/10.1115/ 
IMECE2016-65513. 
C. Colberg Poley and B. Balachandran. Motion analysis of robot arm for obsta- 
cle avoidance. In ASME 2017 International Mechanical Engineering Congress 
and Exposition, volume Volume 4A: Dynamics, Vibration, and Control, pages 
1?9, 11 2017. 10.1115/IMECE2017-71010. URL http://dx.doi.org/10.1115/ 
IMECE2017-71010. 
G. Ricci, F. Trabalzini, M. Faralli, L. D?ascanio, C. Cristi, and E. Molini. Cochlear 
implantation in children with ?charge syndrome?: surgical options and outcomes. 
Eur Arch Otorhinolaryngol, page 489?493, Mar 2013. DOI10.1007/s00405-013 
-2424-1. 
C. Richard, J.N. Fayad, J. Doherty, and F.H. Linthicum. Round window versus 
cochleostomy technique in cochlear implantation. Otology and Neurotology, 33 
 149 
(7):1181?1187, Sep 2012. 10.1097/mao.0b013e318263d56d.  URL https://www 
.ncbi.nlm.nih.gov/pmc/articles/PMC3425957/. 
N. Robbins. Discussion of radiologist understanding of cochleostomy &mastoidec- 
tomy ct scans, Jan 2019. 
R. Sahyouni. Cochlear implants. https://www.khanacademy.org/ 
science/health-and-medicine/nervous-system-and-sensory-infor/ 
sound-audition-topic/v/cochlear-implant, 2014. 
H. W. Sampson, J.L. Montgomery, and G.L. Henryson. Atlas of the human skull. 
Texas A and M University Press, 2007. 
D. Scorza, S. El Hadji, C. Cortes, and A. Bertelsen. Surgical planning assistance 
in keyhole and percutaneous surgery:  A systematic review. Medial Image 
Analysis, pages 1?22, January 2021. URL https://www.researchgate.net/ 
publication/344784764 Surgical planning assistance in keyhole and 
percutaneous surgery A systematic review. 
B. Siciliano,  O. Khatib,  and T. Kro?ger.  Springer  handbook  of  robotics.  Springer., 
2016. 
R. Siegwart, I.R. Nourbakhsh, and D. Scaramuzza. Introduction to autonomous 
mobile robots. MIT Press, 2011. 
M.W. Spong, S. Hutchinson, and M. Vidyasagar. Robot Modeling and Control. J. 
Wiley and Sons, 2012. 
C. Terray. Move your body better. http://breakthroughgym.com/move-your 
-body-better/4614/, Feb 2016. 
R. Kliegman et al., B. Stanton, R. Behrman, and W.E. Nelson. Nelson?s Textbook 
of Pediatrics. Elsevier, 19 edition, 2007. 
A. Tits. Notes for ENEE 664: Optimal Control, volume 1. University of Maryland 
College Park, 2018. URL https://user.eng.umd.edu/~andre/664.pdf. 
Mid Cheshire Hospitals NHS Foundation Trust. https://www.mcht.nhs.uk/ 
information-for-patients/departmentsandservices/audiology/ understanding-
hearing-loss/anatomy-of-the-ear/, 2019. 
G. Wanna, J.Pile, and N. Simaan. Force-based flexible path plans for robotic elec- 
trode insertion. In 2014 IEEE International Conference on Robotics and Automa- 
tion (ICRA), pages 1?8, 05 2014. 10.1109/ICRA.2014.6906625. 
E.W. Weisstein. Logarithmic spiral. http://mathworld.wolfram.com/ 
LogarithmicSpiral.html, 2019. 
WHO. Deafness and hearing loss. https://www.who.int/news-room/fact 
-sheets/detail/deafness-and-hearing-loss,   2019. 
 150 
J. Zhang, J.T. Roland, S. Manolidis, and N. Simaan. Optimal path planning for 
robotic insertion of steerable electrode arrays in cochlear implant surgery. Journal 
of Medical Devices, 3(1):011001, 2009. 10.1115/1.3039513. 
 151