Raycasting With LayerMasks in Unity

Objective: Using a LayerMask with our Raycast to determine which game objects can change color when clicking on them.

When using Raycasts, LayerMasks come in handy as we can provide additional checks to determine which game objects we are able to click on, depending on which layer they are on.

E.g. A great example of this is in shooting games where they have the feature of being able to turn “Friendly Fire” on and off. LayerMasks are one approach to this solution.

In the above example I have 2 different types of game objects — A cube and a capsule. The capsule is simply using a default layer and the cube is using an ‘Enemy’ layer that I have created. And within the Raycast we can specifically tell it which layer to look for.

Getting the Scene Setup

I have simply created a capsule to be used as the ‘Player’ and 2 cubes to be the ‘Enemy’.

I have then created a new layer in the Inspector and named this ‘Enemy’.
Note: The ‘User Layer Number’ next to Enemy — 6. This will come in handy when we jump into our code.

Make sure to set the ‘Enemy’ cubes to the ‘Enemy’ layer.

Building Code

Option 1 — Using a String to Determine LayerMask

This is achieved through the GetMask(“Enemy”) static built-in method.

Option 2— Using Bitwise Operators to Determine LayerMask (more optimised)

There is a more optimised way of approaching this using bitmasks. Bitmasks represent the 32 layers of a LayerMask which are defined as ‘False’ or ‘True’. ‘False’ representing 0 and ‘True’ representing 1 respectively.

E.g. If we wanted to turn the ‘Enemy’ layer mask on (which has a user layer number of 6), we would set bit 6 to 1 (true).

This is achieved through the use of bitwise left shift operator ‘<<’.

1 << 6 

// This essentially means that we are setting bit 6 to the value of 1 (true)

Option 3— Using a LayerMask Serialized Dropdown Field in the Inspector

Our final solution is to create a LayerMask Serialized field in the Inspector so we can click the drop down menu and select ‘Enemy’ and this will be passed into the Physics.Raycast parameter.

Article Links
Layer Masks — Unity Documentation
LayerMask.GetMask() — Unity Documentation
Bitwise Operators
Physics Raycast — Unity Documentation
RayCastHit — Unity Documentation
RayCastHit.point — Unity Documentation
Ray — Unity Documentation
Debug.Drawline — Unity Documentation
Polling Actions — Unity Documentation
ReadValue<> — Unity Documentation
Casting Raycasts from Mouse Position in Unity
Placing Objects via Raycast Hits in Unity

Next Article
“Creating Bullet Holes via Raycasting”