Two-Handed Interactions with Unity XR Interaction Toolkit

How strange would it feel wielding a two-handed object in VR with only one hand using our Oculus (or Meta) or your favorite virtual reality headset? Or something utilitarian like holding a recurve bow on one end and knocking the arrow with the other hand.

To achieve high immersion, the objects in VR experiences have to behave similarly to the user's expectations. This is mandatory for business applications like training simulators.

Oftentimes we need to grab objects with both our hands in VR. The problem is the Unity new input system XR Interaction Toolkit SDK does not have support for this.

So we have to implement it ourselves!

There are a lot of different ways we can interact with an object using two hands. In this tutorial, I’m going to show you how you can override the XRGrabInteractable component to allow us to grab a second anchor point to control the direction of the interactable object.

In this tutorial, I’ll be doing C# Programming and some Vector3 + Quaternion math. Don’t worry about it, I’ll explain along the way.

The method for two-handed interactions has its pros and cons. Before you start implementing this solution be sure to check the drawbacks and consider whether they’ll impact your project.

Pros:

• Easy to set up.
• Small script.

Cons:

• We lose the ability to throw the interactable. This would have to be reimplemented in our script.
• The first grab will always grab the same spot.

Before we get started, make sure you have the XR Interaction Toolkit set up and installed in your Package Manager within Unity editor. If not, follow this guide to do so.

For this tutorial we’re going to build a two-handed water gun. This example project should teach you how to use the technique on your own VR development projects. If you get lost, check out our discord channel with hundreds of other xr developers and designers.

1.Setup Scene

First I will create an interactable object (water gun) out of primitive 3D shapes. This will be the object we are going to grab.

We want to make sure orientation of the water gun is proper. The Z axis (blue arrow) should point forward, and the Y axis (green arrow) should point up.

2. Make it interactable using an Attach Transform.

On the root of the gun, add an XR Grab Interactable Component.

This will automatically add a Rigidbody. Let's test to make sure we can interact with this gun in VR.

We can pick up the gun, but we are not grabbing it by the handle. We can use an Attach Transform to tell the gun where its anchor point is.

Right click on the gun root and create an empty GameObject. Call it Handle Attach.

Position the Handle Attach GameObject where you want to grab the gun from.

It’s important that we do not change the rotation as it will affect how our hand grips the handle.

To tell the XR Grab Interactable to use this Handle Attach, click and drag the Handle Attach into the Attach Transform serialized property.

Now we can see that we are grabbing the water gun from the handle.

This works great for one-handed interactions, but what if we wanted to hold this water gun with two hands? We need to implement our own XR Grab Interactable to achieve this.

3. Inheriting from XR Grab Interactable

Create a script called DoubleXRGrabInteractable. Right click in the Project window and go to Create > C# Script.

Double-Click on the script to open it up in your code editor.

Go ahead and delete the Start and Update methods.

In order for us to inherit all of XRGrabInteractable’s code, we need to import the library into our script. Add using UnityEngine.XR.Interaction.Toolkit;

Next, replace MonoBehaviour with XRGrabInteractable.

Now our script will function exactly like the XRGrabInteractable, however we are able to override its logic and make up our own.

4. Overriding ProcessInteractable

In the scope of the class, type override. Add a space and you will see a list of all the things you can override. ProcessInteractable is the method that will be called every frame to move the interactable into the hand, so let's override it.

So far nothing has changed, this method will be called every frame still, and base.ProcessInteractable(updatePhase); will call the default logic (to move the object to our hand). But we only want it to do the default logic if we are holding this interactable with one hand.

Because we are inheriting from XRGrabInteractable, we have access to most of its variables. One variable we can use is a List of IXRSelectInteractor’s called interactorsSelecting. This is a list of the current number of hands that are grabbing an object.

We can use this list to tell if we are grabbing an object with one or two hands. If we are grabbing it with one hand, we will do the default logic, else we can do our own logic.

One thing to note: This method will be called for multiple Update Phases. For now, let's only implement our movement logic (read more about VR locomotion types here) in a single phase. For instantaneous movement, we will use the Dynamic Update Phase.

To keep things clean, create a new method “ProcessDoubleGrip” and call that method from ProcessInteractable.

5. Understanding the math

ProcessDoubleGrip will only be called every frame where we are holding our weapon with two hands. We have to manually implement the math that causes the weapon’s transform to move between our hands.

XRI uses an “Attach Transform” to specify an anchor point for grabbing an object. When we grab something, there has to be some math to convert the XRIGrabInteractable’s transform around the anchor point. To do this, we get the local direction and rotation from the anchor point to the base, then apply that local direction to the orientation of the hand.

I’ll be using the world “Pose” in this tutorial to represent position and rotation.

For example, If purple represents the transform of the gun and red represents the Attach Transform.

If we just moved the gun to the hand, it would look like this.

We first need to get the local pose from the Attach Transform to the center of the gun.

Then we can apply that local pose to the hand transform’s current pose to get the new pose of the gun. This way, it follows the orientation of the hand.

This is handled for us by the XRGrabInteractable, but it’s good to understand how it works. We need to make up our own logic for grabbing an object with two hands. When operating a two-handed object like this.

We have to decide on how it will work mechanically:

• How do you get the pose of the weapon between our hands?
• How do you apply an offset to make sure the orientation is proper?
• How to you decide the position of the weapon
  • Should it be anchored from the first grip, second grip, or between them?
• What will control the weapon’s upwards direction?

We can get a direction between two Vector3s (both our hands). This is always Target - Source.

We already defined the handle of the gun, so let's also define a position for the grip. (As another Transform).

6. Create a second Attach Transform

Create a Game Object called “Grip Attach” and position it to the grip of the weapon.

We will need a serialized variable to reference this transform from our script.

7. Setup the new script.

Replace the old XRGrabInteractable script with our new DoubleXRGrabInteractable script.

Re-serialize a reference to the handle’s Attach Transform and the Grip’s Attach Transform.

In order to grip the gun with two hands, we have to set Select Mode to Multiple, but Unity doesn’t let us do this. So we will have to do it with code at runtime.

In the script, let's override Awake.

When the game object initializes, set the select mode to multiple.

8. Get the Data

In the ProcessDoubleGrip method, we’ll start by getting access to all the data we need.

• The weapon’s handle and grip transform.
• The transform of both the interacting hands.
• The weapon’s Attach Transform can be acquired using the method GetAttachTransform(null).
• We can use the interactorsSelecting list to get both the interacting hands.

9. Get Position & Rotation

We need to get a direction for the gun using the direction between our hands. Again, the direction between two points is Target - Source. This will give us a vector like this:

We can then use Quaternion.LookRotation to create a rotation that uses that vector as the forward direction.

We need an anchor point for the water gun to move to. For this, let's continue to use the handle (first Attach Transform). First get the direction from the weapon’s handle to the base.

This direction will be in world space, so we need to convert it to local space. We can use transform.InverseTransformDirection().

Now that we have a proper offset from the handle to the base, we just need to apply that offset to our hand’s position.

10. Set Position & Rotation

Let's test this out! Set the position and rotation of the weapon to the values we acquired.

You may notice a couple issues:

• The position of the gun isn’t in the right spot.
• The gun can not be rotated on the forward axis at all.

When we grab the gun with both hands, the weapon’s forward direction will be between our hands, but the positional offset doesn’t take into account this rotation:
firstHand.position + localDirectionFromHandleToBase

We need to change the direction of localDirectionFromHandleToBase to match the new direction of the gun. To do this, we multiply the rotation by the direction.

We should get something like this.

If we want to rotate the gun on the Z axis, we need to decide what controls that rotation.

I personally like to control the rotation with the first interactor. All we have to do is set the upwards direction of the gun to the upwards direction of the hand.

One more issue with this math you will notice: The second Attach Transform is not at the proper position of our hand. It should look something like this:

Well currently we are just setting the forward direction of the gun to the direction between our hands. We need to apply a rotation that will convert the direction between the attach transforms to the new forward direction of the gun.

First get the direction between the Attach Transforms.

Next, we want a rotation that goes from the attach transform direction to the forward direction. (Represented by the green line). Then apply that rotation to the end result.

To do this, we can use Quaternion.FromToRotation().

To apply this rotation, we multiply it by the end result rotation.

Note: When multiplying rotations, it’s important that we multiply them in the proper order.

11. Non-Kinematic Rigidbodies

The interactable should be working properly, but there is a problem for non-kinematic Rigidbodies: The interactable doesn’t drop anymore.

When we first grab an XRGrabInteractable, it performs on-grab logic to set up the interactable’s Rigidbody. Same thing for when we drop the interactable.

An easy way to fix this is to make sure that it only performs the grab logic when we first grab the interactable and only performs the drop logic when we are no longer interacting with it.

We can override these methods!

For Grab, we only want to run the default logic if it’s the first grip.

For Release, we only want to run the default logic if we are not interacting. We could check interactorsSelecting, or use the bool isSelected.

12. Shooting

Typically you will use the OnActivated and OnDeactivated events to start/stop shooting.

If you only want these events to happen when the first interactor pulls the trigger (The hand on the handle of the gun). We can override the methods and use another if statement.

The first interactor is always interactorsSelecting[0].

All we need to do is compare if this first interactor is the current interactor that is calling the method (activating). We will find this data in the ActivateEventArgs interactorObject.

That’s all for this tutorial!

Although we just set up a water gun, the logic we have used can be applied to other sorts of two-handed interactions as well. Stay tuned for more VR tutorials in the future!

Looking to learn the fundamentals of XR development and acquire some of the advanced techniques like two-handed interactions above? Check out the XR Development with Unity course.