2019 is gonna be the year of AR.
Made this little tool with Unity, unbelievable fun and addicting.
It tracks the coloring process live and wrap the real-time texture on the 3D model, and take basic finger gestures for model interactions.
Used Image recognition to trigger AR experience that allows you to explore the new car without being in the store. Experience made in Unity.
It was a pleasure working on Verizon’s new family plan selection site, which is live on its main webpage https://www.verizonwireless.com/plans/unlimited/.
It was a big challenge and great experience exploring creative ways for user interactions on the video banners, plan selections, creative page transitions, physical simulated graphics, css animations as well as applying responsive designs for each element.
I spent couple days digging into web AR research due to some growing requests about making AR experience more accessible and sharable. The result isn't beyond my expectation, with the benefit of no-need-to-install separate AR app, it comes with the scarification of weaker power, instability and attachments with experimental browsers which allows to use the system level ARKit/Core functionality.
Maker based 3D model displaying using AR.js and THREE.js. I designed a marker using the tool with 3 different sizes, to see which one works better. I simply load and place a free car model using THREE.ColladaLoader and tested on 3 phones (Iphone 6S, iphone x, samsung galaxy s7 ). I updated the simple demo on github.
The recognization works the best with smaller size markers, even tracks the rotations of paper while the bigger marker is harder to contain because the whole marker has to be in the camera view to be recognized otherwise it will lose the model, different from Vuforia / image recognization when it would still work even if it detects only parts of the image. The model is laggy and pixelated in the mobile view with iphoneX. In 6s, it refuses to open due to "denied permission".
tons of libraries made it easy to create 3D scenes without a deep knowledge of 3D development or native programming experiences with ARkit/ARCore/Game engines
most libraries are compatible with each other, so can easily integrated with other modern web components, frameworks and server APIs
easy to make sharable gaming / AR experience without app installation as long as it doesn't require heavy modeling and processing
requires low poly assets (wouldn't work well for displaying or interacting products with high details and textures )
user Interface design is very different from the standard web
most browsers and APIs are marked as experimental, work in progress and unstable
no native AR support unless installing external browser or use printed Marker
JS Library: Marker-based web AR technology
by detecting its position in the camera's video stream and laying 3D-models on top of it.
extension for A-Frame and Three.js for marker-based AR scenes
An emerging technology from Mozilla that allows to build web 3D/VR/AR experience with simply HTML and DOMs.
an Entity-Component Framework that built on top of WebGL, Three.js, which provides a declarative, extensible, and composable structure to three.js
it follows the Composition over inheritance principle that allows greater flexibility in defining entities where every object in a scene is an entity (e.g. enemies, bullets, vehicles, etc.).
browser support: WebARonARCore, WebARonARKit
use the power of ARCore (for Android) or ARKit (for IOS) both are native SDK’s
based on three.xr.js
complies with the proposed WebXR Device API using the webxr-polyfill
allows you to move between AR and VR seamlessly
X stands for: anything
gateway which the devices use to communicate with the browser and give it information about their capabilities, so that users/devs are able to use this information and adjust the website to any device
XR Session: a communication channel between the browser and the device
Enable XR applications on the web by allowing pages to do the following:
Detect available VR/AR devices.
Query the devices capabilities.
Poll the device’s position and orientation.
Display imagery on the device at the appropriate frame rate.
This API will take over the current WebVR API and expand it to even more devices
an open-source library that we use to localize it through the phone camera.
Browsers that supports native AR powers and features from ARkit/Core but requires to install separately as a application to view AR contents
WebARonARCore: Experimental web browser by Google with the goal to bringing ARCore to Web
WebARonARKit: Similar as above, for iOS version
WebXR Viewer: Similar as above for Mozilla
An important feature of AR.js is the possibility to use custom markers, the default type is ‘pattern’. I did some research on pattern markers’ limitations, on their size and shape. I learned that:
The maximum resolution of a marker is 16x16 pixels
They must be square in shape
They cannot have white/transparent areas, only light grey (e.g. #F0F0F0)
They cannot contain colors, only black and light grey
They have to contain simple text, like one letter, a number, or a symbol.
You may use this online tool
cant do very complex sides and edges.
check Patt file, if too complex, the marker trainer just generated a bunch of 0's
have an high contrast between the ‘background’ of the marker and the surrounding environment
Google Poly: thousands of models for VR/AR use
One of the biggest wearable tech + web app project for work in 2017 for Nike NBA.
Tap the jersey’s jock tag using your smartphone to activate a real-time personalized NBA experience. Follows player and game status and unlock exclusive gear, playlists and game highlights.
chewy role: front-end(react), svg animations (svg.js), work/award site
The first native iOS app built that allows you to take photo and fetch that “dream sneaker” you see someone wearing on the street or in the store that’s out of stuck!
Object detection, and scanning animations are made in Openframeworks and openCV.
Developed this meme generator for Nike Marathon over a week that's live now on nike.com/nyc
Just much as I love arts, when people all over the world can use what you made to share stories and motivate each other's life, how satisfying is that? :)
What is your breakthrough? Define yours at https://www.nike.com/nyc
whatwillyoubreakthrough.com (site will only be live for one month)
“Am I the fastest dude in the city? Nah, not even close. I’m probably the slowest in the running community in NYC, but I really don’t give a &#@k about that. I got my family, I got my crew, I’ve got all the people I hold dear pushing me every step of every mile.” - @hecisdead . All you’ve got to do is believe. Hector Espinal was never athletic growing up. Now, he’s gearing up for his first marathon to prove anything is possible. Click the link in our bio to see him #breakthrough the doubt.
A post shared by NikeNYC (@nikenyc) on
I've broke down pros and cons to each of the popular libraries in the field of SVG/Canvas/WebGL 2d and 3D, including but not limited to D3.js, Rapheal.js, paper.js, pixi.js, p5.js, three.js, babylon.js, paycanvas.js and voxel.js. I've attached examples and docs and tutorials and hopefully this slide will become a 'go-back-to' place next time when you and I need to look for an animation tool for our next cool website.
Like every Makerfaire in the past 4 years, I'm honored to be able to collaborate with some incredibly talented people on some amazing projects.
This time our booth are located at 3d printing area with ThreeForm Fashion engineer / artist: Aaron Trocola, although our mocap fashion suit design is still in process, we are able to demo the video of the suit-in-process as well as all the other creation Aaron made through this year including customized EEG sensor headset, on Dual-Brain machine for Art-a-Hack, and other 3d printed fashionable pieces, models and masks.
One thing I'm absolutely loving is that seeing so many girls (prob double times than boys) have came to MakerFaire and are so curious and excited about those innovative tech and science subjects - those next generation will eventually change the future definition of Sexy.
The performance requires a high-end GPU system with a recommended (must require or you will wait till you die) Nvidia + CUDA6.5+ backend and optimizer. Since macbooks have replaced all Nvidia Graphic cards to AMD (dunno why), I had to perform the renderings on AWS EC2 gpu servers, until either I get a new gaming computer, or I get tired of this thing.
Video of the collection
Some customized results for videos (GIF version) with optical flows:
15 styles of Shanghai
6 styles of Chewy
Couple months ago I created an interactive and immersive virtual reality experience for Nike / Converse store. Customers are able to design their own shoes and virtually wear them on. I decided to put the brief summary here for future reference.
I did three prototypes including a 1. customizable shoe, 2. an interactive environment based on shoe selection and 3. an animated VR character that changes its shoes based on shoe selection
3D loader formats
Algorithm animation (particles, moving forms)
THREEJS - Pros:
THREEJS - Cons:
UNITY - Pros:
Unity - Cons:
other users’ comments:
“ When your company runs into an issue with Unity, the professional people who you are licensing from, and paying, will help you with it.
When you run into an issue with Three.js or any one of the "extensions" you will probably want / need to use - you may or may not find yourself all alone, digging through the source-code.”
“for lack of a better example, you could think of Unity as a train and three.js as a car. Unity is bigger and heavier but once it gets going, it can carry a lot. Three.js is less equipped, but is convenient and easy to get in and out of.
Threejs is a very amazing tool for creative storytelling, real-time animations and data visuals. It’s totally worth a try if you want to create some stunning interactive visuals for your web-based projects, compatible with other web libraries. However it’s not designed for VR experience, or installations which requires complicated interactions and heavy visual processing power. It’s still in a very early stage, the lack of expertise and updated references could cause some big headache for the beginning learns.
Unity in the other hand is a native environment for games and installations, which means it’s much more powerful, fruitful and flexible in all aspects especially for multi-platform projects including VR experiences. The assets stores and Unity API documentation really helps reducing both art assets or code syntax frustrations, which could easily happen for Threejs if you are not a master of both 3D design and low-level programming. Meanwhile it also takes much longer to learn the interface, discover the capabilities and efficient workflows. From both business or development perspective, it’s a time-consuming investment, but it normally leads to a promising result.
In my opinion Threejs is like a container of goodies, easy to take it with you to everywhere, share with your friends. It’s a simple magic that makes people happy.While Unity is the chocolate factory. It takes lots dive in, but once you know how it works and know what you need to make, you are the god of your own world.
Suggestions for THREEJS beginners:
Other similar frameworks / Engines:
Google Cardboard - Pros:
Google Cardboard - Cons:
GearVR - Pros:
GearVR - Cons:
Notes for controllers
Other similar VR headsets: