UAV Attitude Estimation from Monocular Omnidirectional Events

Abstract

Attitude estimation seeks to calculate the orientation (i.e., pitch and roll) of an object in motion. Determining attitude is crucial for flight, but conventional sensing techniques rely on gyroscopes, accelerometers, and magnetometers whose measurements are prone to drift and require an initial reference orientation. We propose a novel attitude estimation method for drift correction and reference orientation obtention by combining two inspirations from biology: wide-field optics and event sensors. We focus on planar motion with 3 degrees of freedom, and we introduce a simple algorithm that estimates the pitch and roll from a set of omnidirectional events, velocity, and altitude. We show that selecting an annular region of events along the scene horizon yields better results than naively processing a set of omnidirectional events in its entirety, and evaluate the approach using experiments on captured data, showing robust attitude estimation.

I Introduction

Attitude estimation is a crucial task for unmanned aerial vehicles (UAVs). Vision-based attitude estimation methods seek to determine UAV orientation (i.e., pitch and roll) from one or more cameras and are particularly useful as a complement to conventional sensing techniques, which rely on drift-prone gyroscopes, accelerometers, and magnetometers. Correcting the attitude errors induced by this drift can lead to decreased errors for downstream tasks such as position determination and path planning.

UAVs typically already contain one or more cameras as part of their payload for use in path planning, data collection, surveillance, etc. Increasingly, event cameras are included in UAV payloads due to their superior power efficiency, reduced data bandwidth, and low latency when compared to conventional cameras. These qualities are crucial for the future of small and low-power robotics, where size and power constraints are severe [9, 8].

We focus here on one such constrained situation: an agent equipped with a single downward facing event camera equipped with hemispherical optics. Our algorithm estimates the relative angle between the agent and the ground (modeled as simple plane) using only the event-set, velocity, and altitude. Wide-field optics are critical in this scenario, as narrow-field optics would fail whenever the camera passes over a textureless region and whenever the camera’s view direction is far from the agent’s translation direction [2, 6]. Hemispherical optics reduce these failures by aggregating motion from all directions, and by always measuring some viewing rays that are close to the direction of translation.

As depicted in Figure 1, our key observation is that we can drastically reduce the computation required to estimate attitude from hemispherical images by coupling our optics to an event sensor. In particular, we introduce a simple algorithm that estimates pitch and roll from an event-set directly. This is in contrast to prior event-based algorithms which require reference frames from a conventional camera [24] or a base-station camera [21].

Additionally, our use of an event camera provides benefits beyond increased power efficiency and reduced data bandwidth. Event cameras offer incredible speed, easily reaching MHz rates while conventional cameras achieve kHz refresh rates at best. They are also less susceptible to motion blur and more resilient to high dynamic range [9].

In summary, our contributions are:

• an attitude estimation algorithm using only an event-set, velocity, and altitude;

• the evaluation of our algorithm on a macro-scale prototype using off-the-shelf components.

Figure 1: We couple hemispherical optics with a neuromorphic sensor to efficiently estimate attitude of an agent undergoing planar motion. We do so directly from a small event-set, as shown in the inset.

II Related Work

Traditionally, attitude estimation has relied on integrating inertial measurement unit (IMU) data to derive the system’s pitch, roll, and heading. In practice, these IMUs have some amount of bias which induce a significant amount of drift in the resulting integrated measurements. Multiple strategies have been developed to account for this bias, including Kalman filtering [1] and sensor fusion with Global Navigation Satellite Systems (GNSS), as surveyed in [3, 16]. However, these solutions are not always viable for application to UAV systems. UAVs may be in GNSS-denied environments, and Kalman filtering makes assumptions on the system’s linearity and error distribution. One research avenue has been the development of vision-based attitude estimation methods, as surveyed in [19, 12, 4].

Much of this work involves finding the horizon in a scene captured with catadioptric optics. The relative orientation can then be derived by finding the plane that intersects the horizon as projected on the image sphere [17, 13, 7, 14, 15]. Another common strategy is to find vanishing points in a scene based off straight-edged features, then use homography to estimate orientation from those lines. This approach has been demonstrated with wide-field optics [22, 5] as well as narrow-field optics [10, 23]. Other approaches estimate attitude from the amount of ellipse deformation found in stereo catadioptric images [15] or using homography between multiple frames [25].

In contrast, there is a very limited amount of prior work utilizing event sensors. One prior work uses a stationary base-station event camera to estimate the attitude of an UAV fitted with LEDs [21]. Another estimates attitude as a part of a larger motion estimation framework using the eight-point algorithm in conjunction with an additional conventional sensor [24].

As summarized in Table I, prior research utilizes wide-field optics or event cameras, but not both. Our proposed algorithm seeks to fill this gap by harnessing the advantages of wide-field optics and event cameras.

TABLE I: Related Work Summary

Camera Type	Related Work	Algorithm
Conventional Wide-Field1	[17, 13, 7, 14, 15]	Horizon Detection
	[22, 5]	Vanishing Points / Lines
	[15]	Ellipse Deformation
Conventional Narrow-Field	[10, 23]	Vanishing Points / Lines
	[25]	Homography
Event Camera2	[24]	Eight-Point Algorithm
	[21]	LED Tracking
1 Papers [15, 14] utilize stereo catadioptric cameras
2 Paper [24] utilizes both a neuromorphic image sensor and a conventional image sensor

Prior work on vision-based attitude estimation tend to fall into several broad algorithmic categories. Approaches that use wide angle lenses or catadioptric systems tend to focus on detecting the skyline or horizon of the scene [17, 13, 7, 14], though algorithms that estimate attitude from the vanishing points in the scene [22, 5] or ellipse deformation [15] also exist. Work utilizing conventional image sensors with a typical FOV tend to focus on finding vanishing points in the scene [10, 23] or use homography [25] for their estimation. The few prior works that use event cameras either use the eight-point algorithm as a key part of their algorithm [24] or fix the event camera on the ground, relying on an LED tracking system attached to the UAV [21]. In contrast, our proposed algorithm utilizes both an event camera and a wide FOV.

III Attitude Estimation from Monocular Omnidirectional Events

(a)
(b)	(c)

Figure 2: Overview of our hardware setup. (a): Photo of our custom hardware prototype. A DAVIS346 event camera was fitted with a ${195}^{\circ }$ fisheye lens and mounted onto a linear stage with 3D printed fittings. These fittings allowed for independent control of the roll and pitch of the camera. The camera’s axis configuration is shown in the top right. (b): A photo of the checkerboard scene. (c): A photo of the satellite agricultural scene.

As shown in Figure 1, we focus on an omnidirectional monocular event sensor undergoing planar motion. We denote the rotational velocity along the camera axis as $\omega$ (radians per unit time) and the translational velocity along the $x$- and $y$-axis as $u$ and $v$, respectively (meters per unit time). Let there be some static features ${𝐅}_{𝐢}$ lying along a plane in the scene which have strong local contrast, triggering events upon sensor movement. The output from the sensor over time duration $T$ ($t_1=0$ and $t_N=T$) is a group of $N$ events denoted by ${\{x_i,y_i,t_i,p_i\}}_{i=1}^N$—where, for every $i^{\text{th}}$ event, $x_i$ and $y_i$ represent the location on the image sensor, $t_i$ represents the time of occurrence, and $p_i$ represents the polarity. Event polarity is binary and denotes whether the intensity change that triggered an event was positive (increasing intensity) or negative (decreasing intensity). We assume that the change in the depth of a feature from the image sensor is small and negligible over a small time duration $T$. In other words, we assume that the displacement of the sensor with respect to the average depth of the features from the sensor is small over the given time window. The velocity and altitude of the agent is assumed to be obtained from other sensors, such as IMU data, single point-LiDAR, or barometric altimeters.

The goal of our algorithm is to estimate pitch and roll parameters $\theta$ and $\phi$, where $\theta$ is the rotation in radians about the $x$-axis and $\phi$ is the rotation in radians about the $y$-axis, for a given event window. For a set of candidate attitude parameters, we first back-project each event in event window $N$ to its corresponding 3D coordinate along a plane defined by the agent’s depth and candidate attitude parameters. Next we generate a 2D histogram, warping each event from its position at time $t_i$ to the estimated origin location at time $t_1$ and accumulating the number of events at each location. This histogram is dependent on accurate radial depths between the camera center and the 3D coordinates of the scene features and accurate velocities. Accurate attitude parameters lead to converging initial location predictions for each feature, resulting in a sharp histogram. Inaccurate attitude parameters conversely lead to unsharp or distorted histograms, as the initial location predictions for each feature fail to converge. We quantify this sharpness in a loss function that counts the number of high-value bins in a region of interest. This process is repeated for a limited number of candidate attitude parameters, and the candidate with the lowest loss is selected as the final estimation.

In Section III-A, we describe the omnidirectional camera model and calibration. In Section III-B we derive the ordinary differential equations that govern event trajectories on an unit sphere that are induced when the omnidirectional event sensor is undergoing planar motion $(\omega ,u,v)$—i.e., the path which an event generated from a feature takes as a function of time—and provide an efficient implementation of these event trajectories. From here, we can accomplish our goal by finding the set of attitude parameters $(\theta ,\phi )$ that results in maximum alignment of the events along a given set of trajectories, as described in Section III-C.

III-A Omnidirectional Model and Calibration

We used the omnidirectional camera model proposed by Scaramuzza et al [18]. The model uses a Taylor approximation for the mapping between image points, $(x,y)$, and their corresponding unit-length direction vectors for each captured feature ${𝐅}_{𝐢}$ in the scene, ${𝐝}_{𝐢}=(x_s,y_s,z_s)$. The Taylor coefficients are the calibration parameters to be estimated. The projection model is:

$$\left[\begin{array}{c}{x}_s\\ y_s\\ z_s\end{array}\right]=\lambda \left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ f(x^{\prime },y^{\prime })\end{array}\right],\lambda >0,$$ (1)

where $f(x^{\prime },y^{\prime })=a_0+a_1{\rho }_1+\mathrm{\dots }+a_n{\rho }_n$, $\lambda$ is a depth factor, and where $\rho =\sqrt{x^{\prime 2}+y^{\prime 2}}$ is the distance of the pixel from the image center, with $(x^{\prime },y^{\prime })$ related to raw pixel coordinates $(x,y)$ by:

(2)

Here, the affine transformation parameterized by $c$, $d$, and $e$ accounts for geometric distortions and the translation parameterized by $x_c$ and $y_c$ accounts for misalignment errors between the camera and omnidirectional optic’s axis.

In summary, we assume that we know the maps $C(x,y)=(x_s,y_s,z_s)$ and $C^{\prime }(x_s,y_s,z_s)=C^{-1}(x_s,y_s,z_s)=(x,y)$. We performed experimental calibration using frame images from DAVIS346, details of which are explained in Section IV.

III-B Implied Trajectories

Consider an omnidirectional monocular event sensor undergoing a general 6 DOF motion of rotation about an axis $𝐑$ and a translation along a axis $𝐓$. Let there be a captured feature point ${𝐅}_{𝐢}$ at a depth $1/{\sigma }_i$ from the sensor. The rigid motion of the feature ${𝐅}_{𝐢}$ relative to our sensor can be described as a rotation about the axis $𝐑$ and a translation along the axis $𝐓$. The instantaneous velocity of ${𝐅}_{𝐢}$ is

$${𝐅}_{𝐢}=-𝐓-𝐑\times {𝐅}_{𝐢}.$$ (3)

Projecting ${𝐅}_{𝐢}$ onto a unit sphere via spherical perspective projection simply yields the previously defined direction vector:

$${𝐝}_{𝐢}={\sigma }_i{𝐅}_{𝐢}.$$ (4)

We derive the motion field equation by taking the derivative of the projection function in Equation 4, with respect to time and substituting in Equation 3. We obtain the following motion field equation [11]:

$$V({𝐝}_{𝐢})={\sigma }_i((𝐓\cdot {𝐝}_{𝐢}){𝐝}_{𝐢}-𝐓)-𝐑\times {𝐝}_{𝐢}$$ (5)

Using Equation 5, we can find the system of differential equations which govern the event trajectory of a feature point ${𝐅}_{𝐢}$ on a unit sphere, when the omnidirectional monocular event sensor is undergoing planar motion with rotational velocity of $\omega$ (radian per unit time) and translational velocity of $(u,v)$ (radian per unit time). For planar motion, $R=(0,0,\omega )$ and $T/\Vert F_i\Vert =(u,v,0)$. Unlike [11], we assume all features lie on a plane $P$ parameterized by $\theta$, $\phi$, and $Z$, where $Z$ is the normal distance between the camera’s optical axis and the plane, $\theta$ is the relative rotation around the $x$-axis (roll), and $\phi$ is the relative rotation around the $y$-axis (pitch). We now constraint ${\sigma }_i$ to represent the distance between a captured feature ${𝐅}_{𝐢}$ and the plane $P$. For a small time duration $d𝒯=t_N-t_1$ we use the following transformation:

$$\left[\begin{array}{c}{x}_s^{ref}\\ y_s^{ref}\\ z_s^{ref}\end{array}\right]=\left[\begin{array}{c}{x}_s\\ y_s\\ z_s\end{array}\right]-d𝒯\times {\sigma }_i\left[\left(u{x}_s+v{y}_s\right)\left[\begin{array}{c}{x}_s\\ y_s\\ z_s\end{array}\right]-\left[\begin{array}{c}u-\omega y_s\\ v+\omega x_s\\ 0\end{array}\right]\right]$$ (6)

We denote the transformation in Equation 6 as $𝐓$, which transforms an event intersecting a unit sphere at ${𝐝}_{𝐢}=(x_s,y_s,z_s)$ to ${𝐝}_{𝐢}^{\mathrm{𝐫𝐞𝐟}}=(x_s^{ref},y_s^{ref},z_s^{ref})$ for a given attitude estimation for parameters $(\theta ,\phi )$ such that

(7)

III-C Quantifying Alignment

Consider a group of $N$ events triggered in time interval $T(e_i={\{x_i,y_i,t_i,p_i\}}_{i=1}^N)$ collected from an omnidirectional monocular event camera undergoing planar motion. Assume velocities $(u,v,\omega )$ and normal depth $Z$ are obtained from additional sensors. For each candidate set of attitude parameters , we first apply the calibration matrix C as discussed in Section III-A, transforming pixel locations $(x,y)$ to unit-length direction vectors ${𝐝}_{𝐢}$. The radial depths $1/\sigma$ for each event are calculated from $\theta$, $\phi$, and $Z$. This is done by first generating the plane equation then finding the line-plane intersection of each direction vector ${𝐝}_{𝐢}$ with the plane. From here, transformation $𝐓$ is applied as described by Equation 6, obtaining the estimated origin locations ${𝐝}^{\mathrm{𝐫𝐞𝐟}}$ at time $t_1$ for each event. Finally, we apply the inverse calibration matrix $C^{-1}$ to project the scene locations $(x_s^{ref},y_s^{ref},z_s^{ref})$ back to the image plane, as denoted by $(x_s^{ref},y_s^{ref})$. This complete transformation is represented as warp $𝐖$:

(8)

From these warped pixel locations, we construct a 2D histogram using kernel density estimation (KDE). The kernel $K$ is the Gaussian kernel, and the bandwidth $w$ is selected based on the Improved Sheather Jones (ISJ) algorithm [20]. The 2D histogram is defined as the following:

$$H(x_s^{ref},y_s^{ref})=\frac{1}{n{w}^2}\sum _{i=1}^{N}K(\frac{x_s^{ref}-x_{s_i}^{ref}}{w},\frac{y_s^{ref}-y_{s_i}^{ref}}{w}).$$ (9)

The closer the candidate attitude parameters are to the true attitude, the more the warped initial feature locations $(x_s^{ref},y_s^{ref})$ converge, leading to a sharp histogram with high-value bins. As the candidate attitude parameters stray from the ground truth values, the warped locations diverge from one another, leading to an unsharp or distorted histogram with low-value bins. An ideal loss function would be minimized as the histogram sharpens and collates into high-value bins, and a naive solution would be to take the variance of the histogram. However, this does not account for the wide-field optics resulting in a variation in the solid angle for each pixel. In practice, this means that the central portion of the camera’s field of view is overrepresented in the histogram while the periphery is underrepresented. To overcome this, we construct a loss that compares the bin values in a peripheral ring to the overall bin values of the histogram:

$$ℒ(\theta ,\phi )=\sum [{\mathscr{H}}_{masked}>Q_{0.75}(\mathscr{H})],$$ (10)

where ${\mathscr{H}}_{masked}$ is the peripheral ring of the histogram and $Q_{0.75}(\mathscr{H})$ is the third quantile of the 2D histogram.

The peripheral ring mask is generated by first projecting the original event window onto the $x$- and $y$-axes, finding the regions above a threshold of $Q_{0.4}$ along each axis’ projection, then finding the largest connected region to eliminate outliers. These connected regions form a box to inscribe the mask ellipse into. The ring thickness is a hyperparameter chosen from two calibration test cases. Our algorithm in its totality is described in short in Algorithm 1.

Parameters : $({\theta }_c,{\phi }_c)$

Inputs: $e_i,d𝒯,(u,v,\omega ),\varsigma$

1ex

foreach in $({\theta }_c,{\phi }_c)$ do

    foreach $e_i$ do

       ;

    end foreach

   $ℒ(\theta ,\phi )=\sum \left[{\mathscr{H}}_{\text{masked}}>Q_{0.75}(\mathscr{H})\right]$;

end foreach

1exreturn ;

Algorithm 1 Attitude Estimation Algorithm

IV Experiments

The omnidirectional event camera consisted of a ${195}^{\circ }$ fisheye lens with a 4.7mm image circle from Commonlands LLC mounted on a iniVation DAVIS346 DVS event camera. The camera was calibrated as explained in Section III-A from captured checkerboard calibration data captured with the camera’s integrated conventional grayscale image sensor. This was then placed on a custom built linear stage with 3D printed brackets to control the camera angle in relation to the target scene. Two different scenes were used: a checkerboard and satellite imagery of an agricultural field. Photos of the prototype as well as the scenes used are shown in Figure 2.

For the first set of experiments, the camera was placed 0.1m and 0.2m away from the target plane. The camera’s pitch and roll were sequentially positioned at $-{20}^{\circ },-{10}^{\circ },{10}^{\circ },$ and ${20}^{\circ }$; and the camera was moved along the linear stage at $-75$mm/s and $5$mm/s along the camera’s $y-$axis. By performing this experiment for the two different scenes, we collected 32 sets of data. Our algorithm was performed on each set of data and the estimated attitude compared to the ground truth measurements.

Figure 3: Real results from our hardware prototype. The first column shows the histogram generated from the output of the warp transformation described by Equation 8 given the ground truth attitude values. The second column shows the histogram generated with the estimated attitude values. The third column shows the loss plot over a set of candidate attitude parameters, with the ground truth attitude marked with a red dot and the estimated attitude marked with a white cross $(\times )$. The final column shows a representation of the estimated plane orientation overlayed onto of the ground truth orientation.

The procedure for the second set of experiments was similar, with the camera distances, velocities, and target scenes remaining the same. The pitch and roll were sequentially set at $-{30}^{\circ },-{15}^{\circ },0^{\circ },{15}^{\circ },$ and ${30}^{\circ }$, resulting in 23 collected sets of data. After estimated the attitude for each set of data, the result was instead compared to the pitch and roll as estimated from IMU data collected from the camera’s internal IMU.

Example outputs for both experiment sets are shown in Figure 3. As shown in Table II, our algorithm estimated attitude regularly within $5^{\circ }$ accuracy. The IMU comparisons in Table III performed similarly, though the roll estimations exhibited a larger discrepancy. This is due to our experimental setup having motion only along one direction, which resulted in an ambiguous IMU-based roll estimation. In these cases, our algorithm outperforms the IMU-based estimation, as it does not have this ambiguity with one directional motion.

TABLE II: Hardware Attitude Estimation Compared to Ground Truth

MAE	(%)	(%)	(%)	(%)
Experiment Set 1
Combined	18.75	59.38	100	100
Pitch $(\theta )$	31.25	68.75	93.75	100
Roll $(\phi )$	6.25	65.63	93.75	100
Experiment Set 2
Combined	21.74	73.91	100	100
Pitch $(\theta )$	43.48	69.57	100	100
Roll $(\phi )$	47.83	69.57	95.65	95.65

The percentage of estimations within the specified degrees of accuracy compared to the ground truth camera attitude. Experiment Set 1 consisted of 32 experiments ranging from $-{20}^{\circ }$ to ${20}^{\circ }$ for both pitch $\theta$ and roll $\phi$. Experiment Set 2 consisted of 23 experiments ranging from $-{30}^{\circ }$ to ${30}^{\circ }$ for both pitch $\theta$ and roll $\phi$. The ”Combined” row is the Mean Absolute Error (MAE) across both pitch and roll. The ”Pitch” and ”Roll” columns are the MAE for each component separate from the other.

TABLE III: Hardware Attitude Estimation Compared to IMU Estimation

MAE	(%)	(%)	(%)	(%)
Experiment Set 2
Combined	13.04	39.13	69.57	91.30
Pitch $(\theta )$	34.78	65.22	91.30	100
Roll $(\phi )$	8.70	30.43	65.22	78.26

The percentage of estimations within the specified degrees of accuracy compared to camera attitude estimated from IMU data. Experiment Set 2 consisted of 23 experiments ranging from $-{30}^{\circ }$ to ${30}^{\circ }$ for both pitch $\theta$ and roll $\phi$. The ”Combined” row is the MAE across both pitch and roll. The ”Pitch” and ”Roll” columns are the MAE for each component separate from the other.

V Future Work

Our methodology could be extended in several ways. Here we outline a few of these avenues for future work:

Optimization via Stochastic Gradient Descent (SGD): Instead of a grid search, the attitude could be estimated through SGD using automatic differentiation to calculate the gradients. The optimization would be initialized from the previous attitude estimation and be bounded by known UAV parameters such as tilt limit. The confidence of the estimated minima is intimately tied to the curvature of the objective function. This curvature can be quantified by the double derivative of the function at every point:

$$C_{\alpha }=\frac{d^2}{d{\alpha }^2}\left[ℒ(\theta ,\phi )\right]|{}_{\begin{array}{c}\theta ={\theta }^{\ast },\\ \phi ={\phi }^{\ast }\end{array}}\text{ for }\alpha \triangleq (\theta ,\phi ).$$ (11)

Extension to Similar Tasks: Our methodology could be extended to similar tasks such as visual odometry and depth estimation. If another system in the UAV payload captures or estimates agent orientation, we can instead optimize for depth. If another system captures or estimates depth, we can instead estimate the agent’s velocities. The advantage here would remain the same—estimation directly from an event-set, without the need for reference frames or calculating optical flow.

VI Conclusion

Our algorithm introduces a method of solving for pitch and roll directly from an event-set. This provides an additional data point apart from traditional estimations that rely on drift-prone IMU measurements, while also imparting the inherent benefits of speed and power and bandwidth efficiency gained by using an event camera. We demonstrated our algorithm on real data from a hardware prototype, showing close adherence to ground truth measurements and IMU estimations. Additionally, our experiments showed increased robustness over ambiguous IMU-based estimations for cases with one directional motion.