Couple months ago I created an interactive and immersive virtual reality experience for Nike / Converse store. Customers are able to design their own shoes and virtually wear them on. I decided to put the brief summary here for future reference.
I did three prototypes including a 1. customizable shoe, 2. an interactive environment based on shoe selection and 3. an animated VR character that changes its shoes based on shoe selection
Prototype1 - Progress
- Managed to handle basic Threejs environment design
- Experienced with loading different 3D formats and it’s pro & cons
- Designed customizable camera views, geometries, materials, environment mapping
- Created responsive loading system and particle system
- Used mobile orientation control and VR render effect
Prototype1 - Challenges
- Animating particles and models significantly slowed down frame rates.
- Camera view was distorted in the GearVR set.
- Visual background was distracting
- User interaction is limited
Prototype2 - Progress
- Applied clean, optimized 3D canvas with new models
- Built more efficient code flows
- Set first person camera view for immersive viewing experience
- Created a better variety of assets, rigged animations from Blender
- Exported assets as Threejs formats
- Retouched textures from photoshop
- Implemented character animation, camera parenting
- Added more algorithm driven visuals effects
- Added cursor and hovering animation
Prototype2 - Challenges
- Lack of simple and intuitive user experiences
- Lack of loading time
- Oversized model and distracting parts of the model
- No clear instructions and navigation
- No visual feedbacks from interactable objects
- Implement into GearVR
Prototype3 - Progress
- Integrated button click of Google Cardboard
- Improved user experience by building an auto-alignment between the scene and camera
- Highlighted hovering effects on interactable objects and the cursor
- Fixed loading time issue and over-sized model issue.
Prototype3 - Challenges
- Visual sickness from Google Cardboard
- Low model quality
- Implement into GearVR
- Need more code optimization to increase performance
Other possibilities (thoughts)
- Create mirrors around the body to reflect the shoe from different perspective.
- Integrate camera sensors to create AR/mixed reality experience.
- Bringing user interface for customizable shoe design
- Try 3D scan to quickly sketch out realistic 3D assets
- Integrate interactivity into websites using Threejs or Unity Webplayer
- Use shaders ( GPU visual processing ) for responsive animation and simulation instead of writing in javascripts
- Use webar extensions bundle to handle Augmented Reality with Three.js.
REVIEW FOR THREEJS 3D WORKFLOWS:
Basics
- Threejs is fairly straightforward and easy to create basic 3D elements such as:
- scenes, lights, cameras, sky, renderer, shaders, object3D, meshes
3D loader formats
- Collada format contains animation data not accessible for customized materials
- JSON format contains all separate information about the model. It enables changeable materials, skinning, animation etc. But normally has a big size
- Fbx format is light and fast for still mode and a prefered format for many webGL integrated websites, it combines textures and mesh are in one file.
- Wavefront OBJ format is a most common text-based format exported by many modeling packages, it’s simple but limited, containing only geometry data. It needs to load its companion file format MTL file in order to bringing materials
Animation
- In general Threejs is lack of top level organization of animation structures, rather there is a lot of duplication.
- SkinnedMesh - most common animation class, derived from Mesh, adds support for array of Bone and Skeleton
- The AnimationHandler/Animation pair:
- AnimationHandler - a global object for creating and advancing playing Animation
- Animation - a hierarchy of keyframe animation tracks usually used for bones on a skinned mesh, closely linked to AnimationHandler.
- KeyframeAnimation - AnimationHandler independent Keyframe track, only used by Collada example, not compatible with the Animation/AnimationHandler framework
- MorphAnimation - AnimationHandler independent MorphTarget animation, only used by *morphtargets_horse.html examples, not compatible with the Animation/AnimationHandler framework:
- Meshes that combine scheduling of animations
- BlendCharacter - SkinnedMesh + Animation Scheduler (sometimes used with manually animated morph targets
- MorphAnimMesh - single Morph Target animation (for simple repeating things.)
- MorphBlendMesh - multiple Morph Target animation blending (meshes that change between morph target animations.)
Raycaster
- Use 2D mouse position as Raycaster origin, shoot to the direction of 3D world.
- The first intersected object is selected as the interactive object.
- The default ray origin for the mobile is (0,0), which is the center of screen.
Algorithm animation (particles, moving forms)
- Math skill == Animation complexity level
Device Orientation
- Link device orientation event to control camera orientation
- The event listener returns three values when it has found a compatible device – alpha, beta and gamma
- You can also apply a 3D object as a parameter of the event
- The latency will make your eye really painful after a while.
VR Rendering
- Use Stereo effect render, only works for google cardboard right now.
Texts
- First.. it’s better to avoid long portions of text in a VR environment
- Text Geometry works great for easy 3D text rendering, but it generates thousands of vertices and faces and create new material for every string. If you try to load 1000 string it will crash the browser.
- Text Labels: create 2D text container at assigned position that always orthogonal to the camera
- Texture Canvas: draw text on a new canvas element and use it as a texture for a mesh
REVIEW FOR THREEJS vs UNITY
THREEJS - Pros:
- Shows webGL’s great capability without the requirements of learning WebGL
- Great for innovative and interactive web experience
- Designed to make it easy to do basic game-quality 3D graphics.
- Gives a very clean, low level access to all the rendering (projection, animation) capabilities.
- Multiple advanced render controls such as Post-Processing, multipass rendering, and deferred rendering, knowing it well could deliver truly realistic effects
- Instant loads and integrates well into existing website code
- Easy to access through web, users don’t need to download apps which takes massive memories
- Easy to work with other javascript libraries
- Cool examples that covers almost all aspects.
- Open Sources and extensions like THREEx
THREEJS - Cons:
- Early in development, lack of documentation and informative reference
- The framework changes frequently, which leads to lots of outdated solutions
- It’s more of a renderer than a real engine, it will easily reaches its limitations in terms of interaction and visual processing power
- Graphic performances are highly relied on browsers and mobile powers.
- Not compatible for native VR experience.
UNITY - Pros:
- It’s a real engine with complete visual editor/GUI, level designer, texture editor, components systems and physicals systems
- Well documented API and examples
- Stable release and native performance
- Great assets / demo scenes for prototypes or boilerplate
- Supports a wide range of asset formats and converts automatically to optimal formats for the target platform
- Fits in all major media platforms (web, mobile, smartTV, console, VR/AR)
- Big community supports
- Free license that covers the majority of features
Unity - Cons:
- Steeper learning curve
- Frequent involvement of 3D elements (steep learning curve again for 3D design skills otherwise increasing asset cost)
- may overkill if dealing simple 3d integration works on websites.
- Not as web friendly as webGL, not easy to access to the DOM element and other js on the page like Threejs
- Hard to share code and assets, which will be hard to collaborate with other developers
- No source code, you gotta wait for the company to fix bugs and updates
- Paid Pro Desktop version is $75/month, so is iOS and Android version.
other users’ comments:
“ When your company runs into an issue with Unity, the professional people who you are licensing from, and paying, will help you with it.
When you run into an issue with Three.js or any one of the "extensions" you will probably want / need to use - you may or may not find yourself all alone, digging through the source-code.”
“for lack of a better example, you could think of Unity as a train and three.js as a car. Unity is bigger and heavier but once it gets going, it can carry a lot. Three.js is less equipped, but is convenient and easy to get in and out of.
my Summary:
Threejs is a very amazing tool for creative storytelling, real-time animations and data visuals. It’s totally worth a try if you want to create some stunning interactive visuals for your web-based projects, compatible with other web libraries. However it’s not designed for VR experience, or installations which requires complicated interactions and heavy visual processing power. It’s still in a very early stage, the lack of expertise and updated references could cause some big headache for the beginning learns.
Unity in the other hand is a native environment for games and installations, which means it’s much more powerful, fruitful and flexible in all aspects especially for multi-platform projects including VR experiences. The assets stores and Unity API documentation really helps reducing both art assets or code syntax frustrations, which could easily happen for Threejs if you are not a master of both 3D design and low-level programming. Meanwhile it also takes much longer to learn the interface, discover the capabilities and efficient workflows. From both business or development perspective, it’s a time-consuming investment, but it normally leads to a promising result.
In my opinion Threejs is like a container of goodies, easy to take it with you to everywhere, share with your friends. It’s a simple magic that makes people happy.While Unity is the chocolate factory. It takes lots dive in, but once you know how it works and know what you need to make, you are the god of your own world.
Suggestions for THREEJS beginners:
- Most people learn Three.js by studying the examples. But be careful to avoid outdated books and tutorials, and avoid examples/answers from the internet that uses old versions of the library.
- Have basic modeling and animation experience in 3D environment
- Know basic raw webGL knowledge. Would be helpful for Unity too.
- Level up your linear algebra skills.
- Level up your understanding about MVP dimensions (Model View Projection).
Other similar frameworks / Engines:
- Babylon.js http://www.babylonjs.com/
- Cinder https://libcinder.org/
- Unreal Engine https://www.unrealengine.com/
REVIEW FOR VR HEADSETS
Google Cardboard - Pros:
- Cheap and portable
- Easy to setup and integrate with Threejs or other VR platform
- Great for demonstrating prototypes and concepts
Google Cardboard - Cons:
- You need constantly holding it and your eye gets dizzy easily
- It has a limitation of interaction
- To relaunch projects, you need to take the phone out everytime
- Risks of phone falling out
- No tracking sensor, which requires a lot of work in codebase to avoid mismatches of eye-movement and device orientation.
GearVR - Pros:
- Free your hands to use buttons
- Fits to all head sizes and adjustable focus
- The Tracking sensor delivers a more comfortable and intuitive immersive experience
- Able to launch tons of VR app including Oculus apps, Netflix Virtual theater etc, inside the headset once plug in the phone
- Able to use touchpad and gamepad control interaction
GearVR - Cons:
- Requires native app development
- Could be a good amount of investment including the headsets, the phone, and associated products like gamepad, controllers, etc.
- May not get the full experience most out of Gear VR without those controllers.
Notes for controllers
- Gamesir G3s, ( $47.99 ) supports a wide range of other devices.
- SteelSeries Stratus XL ( $99.99 ) is a better designed controller, though you certainly pay for the price and it doesn’t run on rechargeable batteries.
- SteelSeries Free ($30) is rated the best all-around gamepad for the Gear VR.
- Beboncool mini controller ($30) is the best solution for portable gaming.
- Gear VR: What you need to know by Jamie Feltham
Other similar VR headsets:
- HTC Vive
- Oculus Rift
- Sony PlayStation VR
- Microsoft HoloLens
Reference
- 109 Examples for Jos Dirksen’s book: Learning Three.js: The JavaScript 3D Library for WebGL: http://www.smartjava.org/content/all-109-examples-my-book-threejs-threejs-version-r63
- Udacity Interactive 3D Graphics course: https://www.udacity.com/course/interactive-3d-graphics--cs291
- Top Game Development Tools: Pros and Cons: http://www.developereconomics.com/top-game-development-tools-pros-cons/
- Getting started with three.js and WebGL by Jaume Sanchez Elias at JSConf Budapest 2015: https://www.youtube.com/watch?v=HwkGTYRopYg
- Great resource for all VR contents: http://www.vrfocus.com/
- Threejs Texts: It’s 2015 and drawing text is still hard (WebGL, ThreeJS) by Parris Khachi