A guest post by Jan Graf
Last month I tested the Metaio Creator 3.0. Since then I have played around with the software a few more times. What really impressed me was that you can not only track images, but even real objects. The newest method of 3D tracking even works on models without much texture, and in changing light conditions; however, in order to do so you will need a 3-D model in “.obj” format of the object you want to track so that the AR software has a reference model to compare to. I was wondering how you can get such models without building them in a 3D modeling tool. Upon a recommendation I tested the free iOS app 123D Catch from Autodesk. I was told that I would be able to create my own 3-D models from iPhone or iPad photos and process them with the Metaio Creator 3.0. For non-Apple users there is also a web app, where you can upload and convert photos from any device you have. Below I will describe how easy it works and the quality of the results.
Create your own 3-D model using the iOS app 123D Catch
At first you have to download Autodesk’s 123D Catch from the iTunes store for free and install it on your iPhone or iPad. Before you create your first 3-D model, the app will ask you to create an account but that is free of charge as well. Upon initial startup, the software will give a brief instruction that shows you the most important parts of the app in a few simple steps.
To create a 3-D model out of an object you will need many overlapping images. 123D Catch is able to assemble up to 70 individual photos into one 3-D model. Just press the + sign on the upper right corner, next select “Start a New Capture” and shoot a row of at least 20 photos from various angles around your object. In doing so always frame and focus the whole object to get a good result. Some objects require that you move around them a second or a third time and take additional pictures at an angle from above or below to get a full capture of all surfaces needed to build a 3-D model. For example, for chairs you should not forget the underside of the seat. Another example is my attempt to make a 3-D model out of a LEGO plane. That worked only at the second time when I put the plane on a small pedestal to be able to photograph the underside as well. Most of the time, I took the maximum number of photos – better too many than too few!
You can even make a 3-D model out of a person, but of course the person has to stay completely still as long as the pictures are being taken. Transparent objects do not work well and neither do objects with very shiny surfaces. For example, my attempts to create 3-D models out of a glass bottle or a glossy printer were not successful. To achieve better results make sure that there is a contrast between your chosen subject and the background surface. If your subject is small, a colorful tablecloth underneath, a newspaper or some other salient marks around it would provide appropriate reference points for the software. I always put some colored sticky notes with symbols on them around my subjects and took at least 22 pictures. This has proven to be a good method of getting the most consistent captures.
Overall, the quality of the results in my tests varied. Models of people especially often had a few holes or strange deformities (but that can be repaired afterward, as I will explain later on). Only very few tests failed completely, but those failures were probably due to too much backlighting from the sun on some photos. The best 3-D models I achieved were when the object or the person had many details or colors, they were well illuminated, the lighting conditions were constant (not too dark and no backlight) and there were some salient features in the background. The biggest successes were in making a model house, a projector with a matt surface and a little Chinese ornamental figure.
Once all the pictures are taken, they are uploaded to Autodesk’s servers and turned into a 3-D model. The finished model is then sent to your phone. In my tests I had to wait between 15 and 60 minutes.
Process your 3-D models directly from your browser
After creating your 3-D captures, you can repair them with the 123D Catch web app directly from your PC browser. In order to do this go to apps.123dapp.com/catch/, launch the web app and sign in with your account data. Since your model is in the 123D Catch cloud, you can immediately open your project and start editing. Another option is to upload photos from any device to the cloud and let the web app produce a 3-D model out of them. The advantage of this option is that the models are significantly better if the photos were taken with a good camera. The work steps for the repairs afterwards are the same.
As you can see on the picture above, in most cases, a lot of the surroundings of your object get included in your 3D model, or your model might have some defects. That is normal and can easily be corrected. The user interface of the 123D Catch web app is very intuitive. For example, with only a few clicks it is possible to fill holes, lasso or paint select unnecessary parts to delete, smooth the surfaces or plane cut the model to make it flat on the bottom. Just give it a try! Above you can see how your result should look.
In order to use your 3-D model with the Metaio Creator 3.0, you have to save it as an OBJ file: go to “File” in the top left-hand corner and click on “Export OBJ”. As of writing, exporting OBJ files is free, but there apparently this feature will require a premium account in the future.
After exporting your model in .obj format, using Metaio Creator 3.0 you can put your self-created model as 3D content into an AR experience. You can read my experience report about the Metaio Creator 3.0 to see how this works. When loading your model, a warning box might appear, as a 3-D model created using 123D Catch may be very large for rendering on mobile devices. Often it works anyway, but if not, follow the advice of the warning message.
Prepare your 3-D model for object tracking
Another possibility is to use your model as a trackable itself for object tracking. Object tracking means that the AR experience is triggered from the object itself. We have created a 3-D version of the real-world object in order to “teach” Creator to recognize it through the camera view. The technology behind this feature is called edge based tracking. With the Metaio Creator 3.0 you can augment your model with digital content and determine what should happen when real world object is tracked. For example, you can provide additional information about the scanned object in the form of graphics, videos, audio files or links to websites. I tested this feature with a little Chinese figure, as I will show you below.
In order to use your model as a trackable, you may first need to scale up the size of the model (10 times worked for me) and reduce the size of the texture to avoid memory problems on your mobile device. For this I recommend using the free 3-D graphics application Blender and an image editing program such as Microsoft Office Picture Manager or Windows Live Photo Gallery (Windows) or Preview (OS X). I’ll show you how you can do that in just a few steps:
First download the appropriate package of Blender for your platform, install it on your computer and start the application. You will see a mesh with some geometric objects on it. Press the shortcut key ‘A’ twice to select all unnecessary objects, next press ‘Delete’ (’Entf’) to delete them. Then import your 3D model as follows: Click on File > Import > Wavefront (.obj), and choose your 3-D model. Select ‘Y Forward’ and ‘Z Up’ in the bottom left and press ‘Import OBJ’ at the top right of the menu. Afterwards, click on the white circle at the bottom (highlighted on the screenshot below) and select ‘Texture’. Now your model should be textured. You can zoom in and out with the mouse wheel. To open the bar highlighted on the image below right-click on your model and press ‘N’.
If desired, you can change the location and the rotation of our model there. For ‘Scale’ you need to enter 10 in all three lines and press ‘Enter’. After that your model has the appropriate size for object tracking. Then do the following: Press ‘Ctrl’ (’Strg’) + ‘A’ and click on ‘Location’, press ‘Ctrl’ (’Strg’) + ‘A’ and click on ‘Rotation’ and press ‘Ctrl’ (’Strg’) + ‘A’ and click on ‘Scale’ so that in the end there will be a 0 in each field of ‘Location’ and ‘Rotation’ and a 1 in each field of ‘Scale’.
The next step is to export your model. Click on File > Export > Wavefront (.obj), set a tick mark at ‘Selection Only’ on the left, select ‘Y Forward’ and ‘Z Up’ on the bottom left, give your model a name, choose a storage location and finally press ‘Export OBJ’ on the top right.
Lastly, you have to reduce the size of the texture. These image files named tex_0 (sometimes also tex_1) are 4096 x 4096 pixels in size (!). You will need to resize the texture and save it at a resolution of 512 x 512 pixels with the image editing program you prefer. You most likely already have some suitable application installed on your computer.
Finally your 3-D model is ready to be used also for object tracking with the Metaio Creator 3.0 and the AR browser Junaio. I tested the functionality with my Chinese figure.
Create your own AR scenario with your self-created 3-D model
For this I had to open the Metaio Creator 3.0, click on the + icon (to add a trackable) and choose the option ‘3D CAD Model (Beta)’ under the section ‘Object Tracking’. I then opened my 3-D model and chose the viewpoint from where I wanted to start tracking later on. After clicking on ‘Next’, the model is processed. Finally I had to choose the number of edges which are to be observed when tracking and my model was ready to be applied with the Creator. I could now add any digital content to my model. For the test, I put the Metaio man from Creator’s 3D sample content next to my model and viewed the result on my iPhone. (Read my experience report about the Metaio Creator 3.0. for further information) After loading my channel in junaio, a transparent silhouette of my model appeared on my screen. You simply need to match up the silhouette to the original object by pointing your camera to it (object tracking). Once Junaio recognizes your object, your AR scenario will initialize. As you can see below, my result looks fantastic. I tracked my figure and the Metaioman appeared next to it on the screen. So, the self-created models actually work with the Metaio Creator 3.0 very well.
Now it’s your turn: Create your own 3-D models and use them for your own unique AR projects. It is easy and absolutely no developer skills are necessary. I am sure you will be as impressed as I am. I personally think that I will use my new knowledge for some very special Christmas gifts.