Categories
Desert of the Real: Social VR

Unity VR Sync – Avatar Selector and VRChat world creation

Introduction

This week we looked into rigging the Avatars in Unity with VR IK using the Final IK asset. The process was fairly simple, in terms of just assigning the script to the gameObject which also contained the Animator class. This process also involved setting some control objects which would handle any avatar on scene, as you can see below.

Control Cubes in motion.

Dynamic linking in Unity

The idea of learning this, however, was to be able to dynamically link everything on the scene. Why? When someone enters the game as an avatar, the game has to automatically set the controls for the player, as well as for any other player who is instantiated inside the game. When the game is networked, there is no fixed path or reference point to which scripts or objects can refer too. We have to create these BRIDGES. We create these BRIDGES in order to handle these dynamically linked avatar controls.

  • BRIDGE_ controls are created as a template for any VR headset. When any VR avatar is put into the scene, it will subscribe its body position to the controls. They connect when the avatar is instanciated.
  • These bridges allow us to control our bodies.
  • Below you can see a screenshot of the hierarchy. There are three main sections:
Hierarchy View
  • RED: The Avatars, parented under an empty gameObject.
  • YELLOW: The BRIDGES, which connect to parts of the avatar Rig through the VRIK script. (I attached an image below)
  • BLUE: The dynamically linked controls. The bridges will look for these in order to attach to their positions. There is a BRIDGE for each of the three VR_BodyParts.
VRIK settings.

Character Selection

In order to make the option for the user to change avatar through keyboard input, I created 2 scripts: AvatarSelector, which is attached to the AvatarParent I showed in an earlier image. The second script is AvatarListener, which is attached to each avatar(Alvaro, Satomi_1, and Satomi_2), and it’s parameter can be set to Avatar1, Avatar2, OR Avatar3, which correspond to the ENUM keynames defined in the AvatarSelector script. This means that if you press key 1 in your keyboard, you shall see Alvaro. If you press 2, you shall see Satori_1, and if you press 3 you shall see Satori_2( blue and with a party hat.) You can see the video below, as well as each of the scripts.

AvatarSelector.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using TMPro;
using UnityEngine.InputSystem; 
using System;

public enum AVATAR
{
    avatar1,        //0
    avatar2,
    avatar3
}


public class AvatarSelector : MonoBehaviour
{
    // This will be our array of avatars on scene
    AvatarListener[] avatars;

    // This will be our keyboard input
    Keyboard keyboard;

    // The current Avatar is the onw which will we will see and control. By default it is set to avatar1.
    private AVATAR currentAvatar = AVATAR.avatar1;

    // This will display our current character on screen
    public TextMeshProUGUI avatarText;
    void Start()
    {
        // Set our current Keyboard as an input device.
        keyboard = Keyboard.current; 
        if(keyboard != null)
        {
            Debug.Log("Keyboard found!");
        }
        else
        {
            Debug.Log("Keyboard not found :(");
        }

        // Find all of the gameObject with the AvatarListener component attached. Our avatars.
        avatars = FindObjectsOfType<AvatarListener>();

    }

    private void SetAvatarText()
    {
        avatarText.text = ("You are now " + currentAvatar.ToString());

    }

    public void FixedUpdate()
    {
        // Remember which avatar is active at each frame.
        AVATAR lastAvatar = currentAvatar;

        // If you hit the "1" key in the keyboard, select avatar1
        if (keyboard.digit1Key.ReadValue() != 0)
        {
            // Set the current Avatar to avatar1
            currentAvatar = AVATAR.avatar1;
        }
        // If you hit the "1" key in the keyboard, select avatar1
        if (keyboard.digit2Key.ReadValue() != 0)
        {
            // Set the current Avatar to avatar1
            currentAvatar = AVATAR.avatar2;
        }
        // If you hit the "1" key in the keyboard, select avatar1
        if (keyboard.digit3Key.ReadValue() != 0)
        {
            // Set the current Avatar to avatar1
            currentAvatar = AVATAR.avatar3;
        }

        // Loop through each of our avatars.
        foreach ( var avatar in avatars)
        {
            // If the avatar is not the one we selected, set it unactive.
            if(avatar._avatar != currentAvatar )
            {
                avatar.gameObject.SetActive(false);
            }
            // If the avatar is the one we selected, set it active.
            else
            {
                avatar.gameObject.SetActive(true);
            }
        }

        // Print out everytime you change an avatar
        if(currentAvatar != lastAvatar)
        {

            SetAvatarText();
        }

    }

}

AvatarListener.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
/*
 Attach this to each Avatar. The Avatar Selector class will lokk for this script.
*/

public class AvatarListener : MonoBehaviour
{
    //Select the avatar
    public AVATAR _avatar;

}
Categories
Desert of the Real: Social VR

VR Drawing in Unity with the HTC Vive

Made with

  • Unity LTS version 2019.4.11
  • OpenVR package
  • SteamVR asset
  • HTC Vive.

Result:

Video recording of me testing it out.

Assignment

Our assignment was to develop a tilt-brush type experience in Unity.

  • So I created an empty gameObject with the controller handler on it. You can make this gameObject a child of your controller in the XR rig. ControllerHandler.cs on the bottom of the page.
  • I have a couple of Cube Game objects also around the scene, each one of them has the Color Selector script attached to them. ColorSelector.cs on the bottom of the page.
  • And Last, I have a “Brush” prefab, which consists of a Cube with the Brush Handler script attached. I put this brush prefab NEAR the controller object I made earlier in step 1. BrushHandler.cs on the bottom of the page.

While I get Githtub LFS to work properly, you can find the copies of my scripts at the bottom.

  • I had to add the OpenVR package from the package manager.
  • Then went to Project Settings > Player > XR Settings and clicked on the Virtual Reality Supported checkbox.
Player XR Settings for OpenVR to work.
  • Then, I installed the SteamVR asset from the asset store in order to get some prefabs for the Vive Controllers and premade event handlers.
SteamVR Asset. Highly recommended.

After importing

I added the player prefab which came with the SteamVR interaction Demoscene, as well as the Teleport Prefab. Each of them contained the necessary scripting for the HMD + controllers setup and the control-based teleportation, respectively.

The Teleport prefab was interesting. It contained a TeleportArea prefab which had a script with the same name. This object created the active area or plane in which we could teleport on. By default, this included a ‘Locked’ grid with several active hotspots. This meant you can only teleport to the hotspot areas.

This was easy to remove, by deselecting the ‘Locked’ tick box in the TeleportArea script. I then went into the script and disabled the default material.

I got to play.

First I made a ‘UI’ on my left controller, which consisted of little cubes floating around each of a different color. When you touch a cube with the brush in your right hand, your brush changes to that color.

In order to do that, I made a new action in the SteamVR default Action set. The defaults already comes with several actions, such as Teleport, Squeeze, and others. You can find that in Window > SteamVR Input

Creating custom Actions

I created a custom action called DisplayWristColorPicker and DrawBrush, as seen in the image above. Since they would be assigned to the trigger button, with a click setting, I set them as boolean. I would need to set these actions to my controllers in the SteamVR binding UI.

Open binding UI lets you set the custom actions to the hardware, in this case, the Vive controllers.

Assigning the actions to controller inputs.

Then, I went to the trigger of each controller and added a ‘click’ action with my Display Wrist Color Picker.

Creating the click trigger.

These bindings can be seen in JSON format by clicking on the Show Developer Output button.

I then made a script called ViveInput to check if the actions were being registered as I played….and they did!

To check whether the trigger on the controller is being pressed or not, I used the following function:

public void Update()
    {
        //Check if we are pressing the trigger and invoking the Draw Brush Action
        if (SteamVR_Actions.default_DrawBrush.GetState(SteamVR_Input_Sources.Any))
        {
            print("Displaying Brush!");
            
            Draw();
        }

    }

Scripts!

BrushHandler.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class BrushHandler : MonoBehaviour {

    //create a private variable that stores the currentColor

    private Color currentColor;

    //create a public variable that returns the prefab for drawing (hint, prefabs are gameobjects)

    public GameObject brushShape;

    //create a public function that sets up the color of the currentColor (hint, function must have a Color variable)
    public void setBrushColor(Color setColor)
    {
        //Set our material's color whatever color the Color selector is. We invoke this function from the colorSelector.
        transform.GetComponent<MeshRenderer>().material.color = setColor;
    }

    //create a public function that returns the Color of the currentColor
    public Color returnCurrentBrushColor()
    {
        // Returns the applied color when invoked
        return brushShape.transform.GetComponent<MeshRenderer>().material.color;
    }

}

ControllerHandler.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using Valve.VR;

public class ControllerHandler : MonoBehaviour {
    private BrushHandler brushHandler;

    public void Update()
    {
        //Check if we are pressing the trigger and invoking the Draw Brush Action
        if (SteamVR_Actions.default_DrawBrush.GetState(SteamVR_Input_Sources.Any))
        {
            print("Painting with Brush!");
            // Debug.Log(transform.position);
            Draw();
        }

    }
    public void Draw()
    {

        //find the brush controller using findobject of type

        brushHandler = (BrushHandler)FindObjectOfType(typeof(BrushHandler));

        //see which color is currently applied in the brush
        Color brushHandlerColor = brushHandler.returnCurrentBrushColor();

        //update the material of your brush to that color
        brushHandler.GetComponent<MeshRenderer>().material.color = brushHandlerColor;

        //draw our brush here using instantiating

        Instantiate(brushHandler, transform.position, transform.rotation);

    
    }
}

ColorSelector.cs

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class ColorSelector : MonoBehaviour {

    //Create a public Color Type that can be set from the inspector to be used in each menu Item

    public Color colorSelector;
    private void Start()
    {
        //Set our material color to the colorSelector color

        transform.GetComponent<MeshRenderer>().material.color = colorSelector;
    }

    //Hint Look at Color class in Unity SDK

    //Setup Collider Trigger to send the color of this object to the Brush Handler

    private void OnTriggerEnter(Collider other)
    {
        // Find the brush in scene
        BrushHandler brushHandler = (BrushHandler)FindObjectOfType(typeof(BrushHandler));

        // If there is a brush, set our brush's color with the setBrushColor Method
        if (brushHandler)
        {

                other.gameObject.GetComponent<BrushHandler>().setBrushColor(colorSelector);
        }
        
    }

}
Categories
Desert of the Real: Social VR

Sketch – 03 – Digitizing myself

The task for this week was to create our own 3D avatar using the itseez3D software, installed on an iPad using a Structure sensor mark II. The process was straight forward, following the video Igal recommended as a guide.

In a a general sense, You use the strucure camera to make a 3D model including textures. Then, you upload it as a high quality version on to the cloud platform provided by itseez3D. After paying(it cost me $8 US dollars) you can download the model and upload it on to Mixamo , an Adobe own platform with several Unity-ready mocap animations. You will be asked to locate certain parts of your model’s body, and then proceed to download. Later on, there are two manners I found out that worked.

  • The first one was to import some of the downloaded files from Mixamo into Unity. This means only the obj and png texture file. Once imported, I made a new material and added the texture file as the base map.
  • The second method was to import the Mixamo obj file in to blender and export it as an fbx, already containing the material.

Below you can see how creepy my result was at first.

I later realized each animation was tied to the exported model. That means that any animation you want to have with your character you have to select it first in Mixamo before downloading the animation file.

Then, I was blown away by the power of this representation in a simulated environment. It took an hour to make the whole process and make it seem realistic without so much precision… How much does it take to simulate our reality every instant?…. After all of these rhetorical ramblings and thoughts about simulated realities, I only had one solution.

….

Dancing salsa with myself.

Unity

Reality Capture – Photogrammetry Room

So, My computer crashed several times while uploading my model on Unity. The error was not apparent enough for me to solve it at the time, associated with unknown characters in the control script.

Unity error
Unity error

So below I have a video of the photogrammetry resulted using 340 images and Reality Capture.

Thank you for reading.

Categories
Desert of the Real: Social VR

Sketch – 02 – Unity animation and photogrammetry with Reality Capture

This week we learned more about prefabs and Animation timelines in Unity. Though the main focus was avatar kinematic control I took it on a more simpler path, controlling the transform position of a prefab inside the game using the animation timeline.

I also developed the Roll-a Ball game a bit further, adding tiles that change color when the player is on top of them, as well as controlling the instantiation of each tile prefab through a script attached to the TileManager proxy gameObject. You can find the repository with the project and documentation(in-code) here.

Unity

Reality Capture – Photogrammetry Room

Categories
Desert of the Real: Social VR

Sketch 01 – Roll-a-ball game in Unity

This week’s assignment was to get comfortable using Unity by following the Roll-a-ball tutorial.

You can find the code and project in this github repository. I have used Unity before, however, I did have these takeaways from the tutorial:

  • TextMeshPro is considered to be a standard, instead of normal text.
  • Any gameobject with a collider and a rigid body component is considered dynamic. Unity won’t treat it as static or stationary. The pickup obj should not be considered to be static since it will take unity longer to calculate each frame because the cubes are rotating.
  • Kinematic rigidbodies do not react to physics forces.
  • Gravity set to 0 is a half-assed solution, it will still react to physics.
  • On the Unity Menu, go to Window -> TextMesh Pro -> Font Asset creator, and select the font you want to convert. (thanks to totsboy in SO)

I had an issue where my keyboard appeared as unsupported (See image below). However, after starting the project from 0 again it worked out. I reinstalled my keyboard driver also.

Unity keyboard debugger