2019 is gonna be the year of AR.
Made this little tool with Unity, unbelievable fun and addicting.
It tracks the coloring process live and wrap the real-time texture on the 3D model, and take basic finger gestures for model interactions.
Used Image recognition to trigger AR experience that allows you to explore the new car without being in the store. Experience made in Unity.
It was a pleasure working on Verizon’s new family plan selection site, which is live on its main webpage https://www.verizonwireless.com/plans/unlimited/.
It was a big challenge and great experience exploring creative ways for user interactions on the video banners, plan selections, creative page transitions, physical simulated graphics, css animations as well as applying responsive designs for each element.
I spent couple days digging into web AR research due to some growing requests about making AR experience more accessible and sharable. The result isn't beyond my expectation, with the benefit of no-need-to-install separate AR app, it comes with the scarification of weaker power, instability and attachments with experimental browsers which allows to use the system level ARKit/Core functionality.
Maker based 3D model displaying using AR.js and THREE.js. I designed a marker using the tool with 3 different sizes, to see which one works better. I simply load and place a free car model using THREE.ColladaLoader and tested on 3 phones (Iphone 6S, iphone x, samsung galaxy s7 ). I updated the simple demo on github.
The recognization works the best with smaller size markers, even tracks the rotations of paper while the bigger marker is harder to contain because the whole marker has to be in the camera view to be recognized otherwise it will lose the model, different from Vuforia / image recognization when it would still work even if it detects only parts of the image. The model is laggy and pixelated in the mobile view with iphoneX. In 6s, it refuses to open due to "denied permission".
Couple great things:
tons of libraries made it easy to create 3D scenes without a deep knowledge of 3D development or native programming experiences with ARkit/ARCore/Game engines
most libraries are compatible with each other, so can easily integrated with other modern web components, frameworks and server APIs
easy to make sharable gaming / AR experience without app installation as long as it doesn't require heavy modeling and processing
requires low poly assets (wouldn't work well for displaying or interacting products with high details and textures )
user Interface design is very different from the standard web
most browsers and APIs are marked as experimental, work in progress and unstable
no native AR support unless installing external browser or use printed Marker
Libraries / Tools:
JS Library: Marker-based web AR technology
by detecting its position in the camera's video stream and laying 3D-models on top of it.
extension for A-Frame and Three.js for marker-based AR scenes
An emerging technology from Mozilla that allows to build web 3D/VR/AR experience with simply HTML and DOMs.
an Entity-Component Framework that built on top of WebGL, Three.js, which provides a declarative, extensible, and composable structure to three.js
it follows the Composition over inheritance principle that allows greater flexibility in defining entities where every object in a scene is an entity (e.g. enemies, bullets, vehicles, etc.).
browser support: WebARonARCore, WebARonARKit
use the power of ARCore (for Android) or ARKit (for IOS) both are native SDK’s
based on three.xr.js
complies with the proposed WebXR Device API using the webxr-polyfill
allows you to move between AR and VR seamlessly
X stands for: anything
gateway which the devices use to communicate with the browser and give it information about their capabilities, so that users/devs are able to use this information and adjust the website to any device
XR Session: a communication channel between the browser and the device
Enable XR applications on the web by allowing pages to do the following:
Detect available VR/AR devices.
Query the devices capabilities.
Poll the device’s position and orientation.
Display imagery on the device at the appropriate frame rate.
This API will take over the current WebVR API and expand it to even more devices
an open-source library that we use to localize it through the phone camera.
Browsers that supports native AR powers and features from ARkit/Core but requires to install separately as a application to view AR contents
WebARonARCore: Experimental web browser by Google with the goal to bringing ARCore to Web
WebARonARKit: Similar as above, for iOS version
WebXR Viewer: Similar as above for Mozilla
An important feature of AR.js is the possibility to use custom markers, the default type is ‘pattern’. I did some research on pattern markers’ limitations, on their size and shape. I learned that:
The maximum resolution of a marker is 16x16 pixels
They must be square in shape
They cannot have white/transparent areas, only light grey (e.g. #F0F0F0)
They cannot contain colors, only black and light grey
They have to contain simple text, like one letter, a number, or a symbol.
You may use this online tool
cant do very complex sides and edges.
check Patt file, if too complex, the marker trainer just generated a bunch of 0's
have an high contrast between the ‘background’ of the marker and the surrounding environment
Low Poly Asset Libraries:
Google Poly: thousands of models for VR/AR use
Web AR/VR Resources
One of the biggest wearable tech + web app project for work in 2017 for Nike NBA.
Tap the jersey’s jock tag using your smartphone to activate a real-time personalized NBA experience. Follows player and game status and unlock exclusive gear, playlists and game highlights.
chewy role: front-end(react), svg animations (svg.js), work/award site
The first native iOS app built that allows you to take photo and fetch that “dream sneaker” you see someone wearing on the street or in the store that’s out of stuck!
Object detection, and scanning animations are made in Openframeworks and openCV.
Developed this meme generator for Nike Marathon over a week that's live now on nike.com/nyc
Just much as I love arts, when people all over the world can use what you made to share stories and motivate each other's life, how satisfying is that? :)
What is your breakthrough? Define yours at https://www.nike.com/nyc
whatwillyoubreakthrough.com (site will only be live for one month)
“Am I the fastest dude in the city? Nah, not even close. I’m probably the slowest in the running community in NYC, but I really don’t give a &#@k about that. I got my family, I got my crew, I’ve got all the people I hold dear pushing me every step of every mile.” - @hecisdead . All you’ve got to do is believe. Hector Espinal was never athletic growing up. Now, he’s gearing up for his first marathon to prove anything is possible. Click the link in our bio to see him #breakthrough the doubt.
I've broke down pros and cons to each of the popular libraries in the field of SVG/Canvas/WebGL 2d and 3D, including but not limited to D3.js, Rapheal.js, paper.js, pixi.js, p5.js, three.js, babylon.js, paycanvas.js and voxel.js. I've attached examples and docs and tutorials and hopefully this slide will become a 'go-back-to' place next time when you and I need to look for an animation tool for our next cool website.
Like every Makerfaire in the past 4 years, I'm honored to be able to collaborate with some incredibly talented people on some amazing projects.
This time our booth are located at 3d printing area with ThreeForm Fashion engineer / artist: Aaron Trocola, although our mocap fashion suit design is still in process, we are able to demo the video of the suit-in-process as well as all the other creation Aaron made through this year including customized EEG sensor headset, on Dual-Brain machine for Art-a-Hack, and other 3d printed fashionable pieces, models and masks.
One thing I'm absolutely loving is that seeing so many girls (prob double times than boys) have came to MakerFaire and are so curious and excited about those innovative tech and science subjects - those next generation will eventually change the future definition of Sexy.
The performance requires a high-end GPU system with a recommended (must require or you will wait till you die) Nvidia + CUDA6.5+ backend and optimizer. Since macbooks have replaced all Nvidia Graphic cards to AMD (dunno why), I had to perform the renderings on AWS EC2 gpu servers, until either I get a new gaming computer, or I get tired of this thing.
Video of the collection
Some customized results for videos (GIF version) with optical flows:
15 styles of Shanghai
6 styles of Chewy
Couple months ago I created an interactive and immersive virtual reality experience for Nike / Converse store. Customers are able to design their own shoes and virtually wear them on. I decided to put the brief summary here for future reference.
I did three prototypes including a 1. customizable shoe, 2. an interactive environment based on shoe selection and 3. an animated VR character that changes its shoes based on shoe selection
Prototype1 - Progress
- Managed to handle basic Threejs environment design
- Experienced with loading different 3D formats and it’s pro & cons
- Designed customizable camera views, geometries, materials, environment mapping
- Created responsive loading system and particle system
- Used mobile orientation control and VR render effect
Prototype1 - Challenges
- Animating particles and models significantly slowed down frame rates.
- Camera view was distorted in the GearVR set.
- Visual background was distracting
- User interaction is limited
Prototype2 - Progress
- Applied clean, optimized 3D canvas with new models
- Built more efficient code flows
- Set first person camera view for immersive viewing experience
- Created a better variety of assets, rigged animations from Blender
- Exported assets as Threejs formats
- Retouched textures from photoshop
- Implemented character animation, camera parenting
- Added more algorithm driven visuals effects
- Added cursor and hovering animation
Prototype2 - Challenges
- Lack of simple and intuitive user experiences
- Lack of loading time
- Oversized model and distracting parts of the model
- No clear instructions and navigation
- No visual feedbacks from interactable objects
- Implement into GearVR
Prototype3 - Progress
- Integrated button click of Google Cardboard
- Improved user experience by building an auto-alignment between the scene and camera
- Highlighted hovering effects on interactable objects and the cursor
- Fixed loading time issue and over-sized model issue.
Prototype3 - Challenges
- Visual sickness from Google Cardboard
- Low model quality
- Implement into GearVR
- Need more code optimization to increase performance
Other possibilities (thoughts)
- Create mirrors around the body to reflect the shoe from different perspective.
- Integrate camera sensors to create AR/mixed reality experience.
- Bringing user interface for customizable shoe design
- Try 3D scan to quickly sketch out realistic 3D assets
- Integrate interactivity into websites using Threejs or Unity Webplayer
- Use webar extensions bundle to handle Augmented Reality with Three.js.
REVIEW FOR THREEJS 3D WORKFLOWS:
- Threejs is fairly straightforward and easy to create basic 3D elements such as:
- scenes, lights, cameras, sky, renderer, shaders, object3D, meshes
3D loader formats
- Collada format contains animation data not accessible for customized materials
- JSON format contains all separate information about the model. It enables changeable materials, skinning, animation etc. But normally has a big size
- Fbx format is light and fast for still mode and a prefered format for many webGL integrated websites, it combines textures and mesh are in one file.
- Wavefront OBJ format is a most common text-based format exported by many modeling packages, it’s simple but limited, containing only geometry data. It needs to load its companion file format MTL file in order to bringing materials
- In general Threejs is lack of top level organization of animation structures, rather there is a lot of duplication.
- SkinnedMesh - most common animation class, derived from Mesh, adds support for array of Bone and Skeleton
- The AnimationHandler/Animation pair:
- AnimationHandler - a global object for creating and advancing playing Animation
- Animation - a hierarchy of keyframe animation tracks usually used for bones on a skinned mesh, closely linked to AnimationHandler.
- KeyframeAnimation - AnimationHandler independent Keyframe track, only used by Collada example, not compatible with the Animation/AnimationHandler framework
- MorphAnimation - AnimationHandler independent MorphTarget animation, only used by *morphtargets_horse.html examples, not compatible with the Animation/AnimationHandler framework:
- Meshes that combine scheduling of animations
- BlendCharacter - SkinnedMesh + Animation Scheduler (sometimes used with manually animated morph targets
- MorphAnimMesh - single Morph Target animation (for simple repeating things.)
- MorphBlendMesh - multiple Morph Target animation blending (meshes that change between morph target animations.)
- Use 2D mouse position as Raycaster origin, shoot to the direction of 3D world.
- The first intersected object is selected as the interactive object.
- The default ray origin for the mobile is (0,0), which is the center of screen.
Algorithm animation (particles, moving forms)
- Math skill == Animation complexity level
- Link device orientation event to control camera orientation
- The event listener returns three values when it has found a compatible device – alpha, beta and gamma
- You can also apply a 3D object as a parameter of the event
- The latency will make your eye really painful after a while.
- Use Stereo effect render, only works for google cardboard right now.
- First.. it’s better to avoid long portions of text in a VR environment
- Text Geometry works great for easy 3D text rendering, but it generates thousands of vertices and faces and create new material for every string. If you try to load 1000 string it will crash the browser.
- Text Labels: create 2D text container at assigned position that always orthogonal to the camera
- Texture Canvas: draw text on a new canvas element and use it as a texture for a mesh
REVIEW FOR THREEJS vs UNITY
THREEJS - Pros:
- Shows webGL’s great capability without the requirements of learning WebGL
- Great for innovative and interactive web experience
- Designed to make it easy to do basic game-quality 3D graphics.
- Gives a very clean, low level access to all the rendering (projection, animation) capabilities.
- Multiple advanced render controls such as Post-Processing, multipass rendering, and deferred rendering, knowing it well could deliver truly realistic effects
- Instant loads and integrates well into existing website code
- Easy to access through web, users don’t need to download apps which takes massive memories
- Cool examples that covers almost all aspects.
- Open Sources and extensions like THREEx
THREEJS - Cons:
- Early in development, lack of documentation and informative reference
- The framework changes frequently, which leads to lots of outdated solutions
- It’s more of a renderer than a real engine, it will easily reaches its limitations in terms of interaction and visual processing power
- Graphic performances are highly relied on browsers and mobile powers.
- Not compatible for native VR experience.
UNITY - Pros:
- It’s a real engine with complete visual editor/GUI, level designer, texture editor, components systems and physicals systems
- Well documented API and examples
- Stable release and native performance
- Great assets / demo scenes for prototypes or boilerplate
- Supports a wide range of asset formats and converts automatically to optimal formats for the target platform
- Fits in all major media platforms (web, mobile, smartTV, console, VR/AR)
- Big community supports
- Free license that covers the majority of features
Unity - Cons:
- Steeper learning curve
- Frequent involvement of 3D elements (steep learning curve again for 3D design skills otherwise increasing asset cost)
- may overkill if dealing simple 3d integration works on websites.
- Not as web friendly as webGL, not easy to access to the DOM element and other js on the page like Threejs
- Hard to share code and assets, which will be hard to collaborate with other developers
- No source code, you gotta wait for the company to fix bugs and updates
- Paid Pro Desktop version is $75/month, so is iOS and Android version.
other users’ comments:
“ When your company runs into an issue with Unity, the professional people who you are licensing from, and paying, will help you with it.
When you run into an issue with Three.js or any one of the "extensions" you will probably want / need to use - you may or may not find yourself all alone, digging through the source-code.”
“for lack of a better example, you could think of Unity as a train and three.js as a car. Unity is bigger and heavier but once it gets going, it can carry a lot. Three.js is less equipped, but is convenient and easy to get in and out of.
Threejs is a very amazing tool for creative storytelling, real-time animations and data visuals. It’s totally worth a try if you want to create some stunning interactive visuals for your web-based projects, compatible with other web libraries. However it’s not designed for VR experience, or installations which requires complicated interactions and heavy visual processing power. It’s still in a very early stage, the lack of expertise and updated references could cause some big headache for the beginning learns.
Unity in the other hand is a native environment for games and installations, which means it’s much more powerful, fruitful and flexible in all aspects especially for multi-platform projects including VR experiences. The assets stores and Unity API documentation really helps reducing both art assets or code syntax frustrations, which could easily happen for Threejs if you are not a master of both 3D design and low-level programming. Meanwhile it also takes much longer to learn the interface, discover the capabilities and efficient workflows. From both business or development perspective, it’s a time-consuming investment, but it normally leads to a promising result.
In my opinion Threejs is like a container of goodies, easy to take it with you to everywhere, share with your friends. It’s a simple magic that makes people happy.While Unity is the chocolate factory. It takes lots dive in, but once you know how it works and know what you need to make, you are the god of your own world.
Suggestions for THREEJS beginners:
- Most people learn Three.js by studying the examples. But be careful to avoid outdated books and tutorials, and avoid examples/answers from the internet that uses old versions of the library.
- Have basic modeling and animation experience in 3D environment
- Know basic raw webGL knowledge. Would be helpful for Unity too.
- Level up your linear algebra skills.
- Level up your understanding about MVP dimensions (Model View Projection).
Other similar frameworks / Engines:
- Babylon.js http://www.babylonjs.com/
- Cinder https://libcinder.org/
- Unreal Engine https://www.unrealengine.com/
REVIEW FOR VR HEADSETS
Google Cardboard - Pros:
- Cheap and portable
- Easy to setup and integrate with Threejs or other VR platform
- Great for demonstrating prototypes and concepts
Google Cardboard - Cons:
- You need constantly holding it and your eye gets dizzy easily
- It has a limitation of interaction
- To relaunch projects, you need to take the phone out everytime
- Risks of phone falling out
- No tracking sensor, which requires a lot of work in codebase to avoid mismatches of eye-movement and device orientation.
GearVR - Pros:
- Free your hands to use buttons
- Fits to all head sizes and adjustable focus
- The Tracking sensor delivers a more comfortable and intuitive immersive experience
- Able to launch tons of VR app including Oculus apps, Netflix Virtual theater etc, inside the headset once plug in the phone
- Able to use touchpad and gamepad control interaction
GearVR - Cons:
- Requires native app development
- Could be a good amount of investment including the headsets, the phone, and associated products like gamepad, controllers, etc.
- May not get the full experience most out of Gear VR without those controllers.
Notes for controllers
- Gamesir G3s, ( $47.99 ) supports a wide range of other devices.
- SteelSeries Stratus XL ( $99.99 ) is a better designed controller, though you certainly pay for the price and it doesn’t run on rechargeable batteries.
- SteelSeries Free ($30) is rated the best all-around gamepad for the Gear VR.
- Beboncool mini controller ($30) is the best solution for portable gaming.
- Gear VR: What you need to know by Jamie Feltham
Other similar VR headsets:
- HTC Vive
- Oculus Rift
- Sony PlayStation VR
- Microsoft HoloLens
- Udacity Interactive 3D Graphics course: https://www.udacity.com/course/interactive-3d-graphics--cs291
- Top Game Development Tools: Pros and Cons: http://www.developereconomics.com/top-game-development-tools-pros-cons/
- Getting started with three.js and WebGL by Jaume Sanchez Elias at JSConf Budapest 2015: https://www.youtube.com/watch?v=HwkGTYRopYg
- Great resource for all VR contents: http://www.vrfocus.com/
- Threejs Texts: It’s 2015 and drawing text is still hard (WebGL, ThreeJS) by Parris Khachi