Some time ago I needed to solve analytically the intersection of a ray and a cone. I was surprised to see that there are not that many resources available; there are some, but not nearly as many as on the intersection of a ray and a sphere for example. Add to it that they all use their own notation and that I lack math exercise, after a bit of browsing I decided I needed to write a proof by myself to get a good grasp of the result.

So here goes, the solution to the intersection of a ray and a cone, in vector notation.

1. We define a ray with its origin $O$ and its direction as a unit vector $\hat{D}$.
   Any point $X$ on the ray at a signed distance $t$ from the origin of the ray verifies: $\overrightarrow{X}=\overrightarrow{O}+t\overrightarrow{D}$.
   When $t$ is positive $X$ is in the direction of the ray, and when $t$ is negative $X$ is in the opposite direction.
2. We define a cone with its tip $C$, its axis as a unit vector $\hat{V}$ in the direction of increasing radius, and $\theta$ the half angle between the axis and the surface.
   Any point $X$ on the cone verifies: $(\overrightarrow{X}–\overrightarrow{C})\cdot \overrightarrow{V}=\Vert \overrightarrow{X}–\overrightarrow{C}\Vert \cos \theta$
3. Finally we define $P$ the intersection or the ray and the cone, and which we are interested in finding.

$P$ verifies both equations, so we can write:

$$\{\begin{array}{l}\overrightarrow{P}=\overrightarrow{O}+t\overrightarrow{D}\\ \frac{\overrightarrow{P}–\overrightarrow{C}}{\Vert \overrightarrow{P}–\overrightarrow{C}\Vert }\overrightarrow{V}=\cos \theta \end{array}$$

We can multiply the second equation by itself to work with it, then reorder things a bit.

$$\{\begin{array}{l}\overrightarrow{P}=\overrightarrow{O}+t\overrightarrow{D}\\ \frac{((\overrightarrow{P}–\overrightarrow{C})\cdot \overrightarrow{V}{)}^2}{(\overrightarrow{P}–\overrightarrow{C})\cdot (\overrightarrow{P}–\overrightarrow{C})}={\cos }^2\theta \end{array}$$

$$\{\begin{array}{l}\overrightarrow{P}=\overrightarrow{O}+t\overrightarrow{D}\\ ((\overrightarrow{P}–\overrightarrow{C})\cdot \overrightarrow{V}{)}^2–(\overrightarrow{P}–\overrightarrow{C})\cdot (\overrightarrow{P}–\overrightarrow{C}){\cos }^2\theta =0\end{array}$$

Remember the mouthful earlier about $\hat{V}$ being in the direction of increasing radius? By elevating $\cos \theta$ to square, we’re making negative values of $\cos$ positive: values of $\theta$ beyond 90° become indistinguishable from values below 90°. This has the side effect of turning it into the equation of not one, but two cones sharing the same axis, tip and angle, but in opposite directions. We’ll fix that later.

We replace $\overrightarrow{P}$ with $\overrightarrow{O}+t\overrightarrow{D}$ and work the equation until we get a good old quadratic function that we can solve.

$$((\overrightarrow{O}+t\overrightarrow{D}–\overrightarrow{C})\cdot \overrightarrow{V}{)}^2–(\overrightarrow{O}+t\overrightarrow{D}–\overrightarrow{C})\cdot (\overrightarrow{O}+t\overrightarrow{D}–\overrightarrow{C}){\cos }^2\theta =0$$

$$((t\overrightarrow{D}+\overrightarrow{CO})\cdot \overrightarrow{V}{)}^2–(t\overrightarrow{D}+\overrightarrow{CO})\cdot (t\overrightarrow{D}+\overrightarrow{CO}){\cos }^2\theta =0$$

$$(t\overrightarrow{D}\cdot \overrightarrow{V}+\overrightarrow{CO}\cdot \overrightarrow{V}{)}^2–(t^2\cancel{\overrightarrow{D}\cdot \overrightarrow{D}}+2t\overrightarrow{D}\cdot \overrightarrow{CO}+\overrightarrow{CO}\cdot \overrightarrow{CO}){\cos }^2\theta =0$$

$$(t^2(\overrightarrow{D}\cdot \overrightarrow{V}{)}^2+2t(\overrightarrow{D}\cdot \overrightarrow{V})(\overrightarrow{CO}\cdot \overrightarrow{V})+(\overrightarrow{CO}\cdot \overrightarrow{V}{)}^2)–(t^2+2t\overrightarrow{D}\cdot \overrightarrow{CO}+\overrightarrow{CO}\cdot \overrightarrow{CO}){\cos }^2\theta =0$$

$$t^2(\overrightarrow{D}\cdot \overrightarrow{V}{)}^2+2t(\overrightarrow{D}\cdot \overrightarrow{V})(\overrightarrow{CO}\cdot \overrightarrow{V})+(\overrightarrow{CO}\cdot \overrightarrow{V}{)}^2–t^2{\cos }^2\theta –2t\overrightarrow{D}\cdot \overrightarrow{CO}{\cos }^2\theta –\overrightarrow{CO}\cdot \overrightarrow{CO}{\cos }^2\theta =0$$

Reorder a bit:

$$t^2((\overrightarrow{D}\cdot \overrightarrow{V}{)}^2–{\cos }^2\theta )+2t((\overrightarrow{D}\cdot \overrightarrow{V})(\overrightarrow{CO}\cdot \overrightarrow{V})–\overrightarrow{D}\cdot \overrightarrow{CO}{\cos }^2\theta )+(\overrightarrow{CO}\cdot \overrightarrow{V}{)}^2–\overrightarrow{CO}\cdot \overrightarrow{CO}{\cos }^2\theta =0$$

There we go, we have our $a{t}^2+bt+c=0$ equation, with:

$$\{\begin{array}{l}a=(\overrightarrow{D}\cdot \overrightarrow{V}{)}^2–{\cos }^2\theta \\ b=2((\overrightarrow{D}\cdot \overrightarrow{V})(\overrightarrow{CO}\cdot \overrightarrow{V})–\overrightarrow{D}\cdot \overrightarrow{CO}{\cos }^2\theta )\\ c=(\overrightarrow{CO}\cdot \overrightarrow{V}{)}^2–\overrightarrow{CO}\cdot \overrightarrow{CO}{\cos }^2\theta \end{array}$$

From there, you know the drill: calculate the determinant $\mathrm{\Delta }=b^2–4ac$ then depending on its value:

• If $\mathrm{\Delta }<0$, the ray is not intersecting the cone.
• If $\mathrm{\Delta }=0$, the ray is intersecting the cone once at $t=\frac{-b}{2a}$.
• If $\mathrm{\Delta }>0$, the ray is intersecting the cone twice, at $t_1=\frac{-b–\sqrt{\mathrm{\Delta }}}{2a}$ and $t_2=\frac{-b+\sqrt{\mathrm{\Delta }}}{2a}$.

But wait! We don’t have one cone but two, so we have to reject solutions that intersect with the shadow cone. $P$ must still verify $\frac{\overrightarrow{P}–\overrightarrow{C}}{\Vert \overrightarrow{P}–\overrightarrow{C}\Vert }\overrightarrow{V}=\cos \theta$, or simply, if $\theta <90°$: $(\overrightarrow{P}–\overrightarrow{C})\cdot \overrightarrow{V}>0$.

Note that there is also the corner case of the ray tangent to the cone and having an infinity of solutions to consider. I’ve completely swept it under the rug since it doesn’t matter in the context I was, but if it does to you, you’ve been warned about it. Also remember to check the sign of $t$ to know whether $P$ is in the direction of the ray. You may need to determine which of $t_1$ or $t_2$ you want to use, which depends on your use case. For example is your ray origin inside or outside of the cone?

Now for a little sanity test, let’s consider the corner case $C=O$, where the ray origin is the tip of the cone (thanks Rubix for the suggestion!). We have $b=0$ and $c=0$ thus $\mathrm{\Delta }=0$ and $t=\frac{-b}{2a}=0$ which is the expected result.

I also tried the cases $\theta =0$ and $\theta =\pi /2$, but expanding $\mathrm{\Delta }$ proved too tedious to proceed to the end. So this is left as an exercise, as they say. :)

Finally, to demonstrate that the result is indeed correct, here is a glorious ray traced cone scene on ShaderToy:

I hope this can prove useful to others too.

Oh, and Happy New Year by the way!