Only 30 seconds Concept Video – dive in!

If you are interested in our Selfrionette project, please read the background information below.

Selfrionette: A New VR Controller for Manipulating Full-Body Avatars with Fingertip Force

Overview

Selfrionette is an innovative controller that enables users to manipulate a full-body avatar in VR with fingertip force input. This system overcomes physical and spatial constraints while achieving diverse and highly dynamic haptic interactions.

Interaction with virtual objects using Selfrionette.

Key Features

1. Fingertip Force Input

Selfrionette uses seven single-beam load cells per hand to measure fingertip forces. These sensors are embedded in a spherical casing, allowing for natural hand postures during operation. Each load cell can detect forces of up to 20kg, converting them into digital signals at 80Hz via an HX711 chip, which are then read by a microcontroller.

Degrees of Freedom (DoF):

Thumb and Index Finger: 3 DoF each (up/down, forward/backward, left/right), corresponding to arm and leg movements.
Pinky Finger: 1 DoF (pressing), corresponding to object grasping.

Using both hands, users can achieve up to 14 DoF for full-body control.

2. Translating Force to Motion

Forces measured at the fingertips are converted into avatar movements in VR. This translation is achieved in real-time using inverse kinematics (IK).

Avatar Motion Generation Equation

The avatar’s motion is described as follows:

$$ F_v = \alpha F_l \quad \text{and} \quad m_v \ddot{x} + c \dot{x} + k(x - x_0) = F_v $$

$F_l$ : Force applied to the load cell
$F_v$ : Force acting on the target point (limb endpoint) in the virtual space
$m_v$ : Virtual mass
$c, k$ : Damping and spring constants
$x_0$ : Avatar limb’s initial position

This model allows for intuitive and highly responsive control based on fingertip force input.

3. Haptic Feedback Representation

In addition to motion generation, Selfrionette incorporates the simulation of haptic properties (e.g., weight, friction, elasticity) to enhance the realism of virtual object interactions.

Additional Forces for Haptic Feedback

The motion equation for generating haptic feedback is expressed as follows:

$$ m_v \ddot{x} + c \dot{x} + k(x - x_0) = F_v + f_m $$

$f_m$ : Additional force representing haptic properties.

Each haptic property is implemented as follows:

1. Weight

Weight is represented by incorporating acceleration and gravity:

$$ f_m = -m \ddot{x} - m g $$

$m$ : Virtual object’s mass
$g$ : Gravitational acceleration

2. Friction

Surface friction is modeled as:

$$ f_m = \begin{cases} -f_p, & |f_p| \leq \mu |f_n| \ (\text{static friction}) \\ -\mu' |f_n| \cdot \text{dir}(f_p), & |f_p| > \mu |f_n| \ (\text{kinetic friction}) \end{cases} $$

$f_p$ : Parallel force
$f_n$ : Normal force
$\mu, \mu'$ : Static and kinetic friction coefficients

3. Compliance

Elasticity is expressed using spring properties:

$$ f_m = -k (x - x_c) $$

$x_c$ : Contact point
$k$ : Spring constant

This allows users to feel soft objects and elastic materials realistically.

Publications

UIST 2024

Title: Selfrionette: A Fingertip Force-Input Controller for Continuous Full-Body Avatar Manipulation and Diverse Haptic Interactions
Authors: Takeru Hashimoto, Yutaro Hirao
Paper URL

VRSJ 2024

Title: Selfrionette: Realizing Full-Body Avatar Manipulation and Diverse Haptic Interactions with Fingertip Force Input
Authors: Yutaro Hirao, Takeru Hashimoto
Paper URL

Media Coverage

Denpa Shimbun

Creating “Sensory Experiences”: Avatar Operation and Haptic Reproduction with Selfrionette

Selfrionette