[MUSIC PLAYING] SPEAKER 1: Hi there. My name is Jacob, and I work as a Developer Relations Engineer at Epic Games. The Unreal Engine 5 has been officially released, and I want to take this opportunity to show you

how you can create a 3D model in reality capture and use it as a Nanite mesh in UE5. Nanite is a new virtualized geometry system that enables us to use increased geometric complexity, higher triangle counts, and higher object counts.

This was already demonstrated in the Valley of the Ancients demo with the Unreal Engine 5 Early Access. In this tutorial, I will reconstruct the model of a wooden tower and import it into the medieval game environment created by my colleagues.

Then I'll be able to enjoy this playable experience with my custom asset from reality capture. In fact, all of the Megascan assets used in this beautiful medieval village were once captured and processed by our very talented artists. I was lucky enough to stumble upon this tower

during a walk near a forest. And I immediately thought that it would nicely fit this environment. You might think that I used a drone or proprietary pixel-scanning gear, but I only used my personal 24-megapixel full-frame camera with a 24-millimeter prime lens and a 4-meter-long pole

for attaching the camera. I used the pole to reach the upper parts of the tower's roof. I paired my phone with the camera to control the shutter. Altogether, I captured over 900 images, but not all of them

were aligned. In the end, it wasn't a problem because all parts of the tower were successfully reconstructed. In RealityCapture, I'll use a standard workflow. I'll import the images and use the Detect Markers tool to automate the process of placing control points.

Then I'll import a distance definition from a text file to properly scale the scene. This is followed by the image alignment, setting up the ground plane, setting up the reconstruction region, model reconstruction, model cleanup, texturing, and simplification,

and the texture reprojection. I'll export the model with 10 million polygons as an FBX file using the Unreal Engine export preset. Then I'll import it into Unreal. As I mentioned, I'll be using the medieval game environment that is free to download from the Unreal Engine Marketplace.

In Unreal Engine 5, I'll import the model with textures and build Nanite. I'll add some basic functionality to the generated material, create a material instance, and apply it to the model. Finally, I'll place the model in the level,

and tweak the material instance so the model blends in with the environment more naturally. All right, let's get started. Here I am in RealityCapture. And, as always, the first thing I need to do is import the images.

I'll open my file explorer window. Here's a folder containing all the images. But this time I won't import all of the images at once. I will open the folder and select the first six images that contain only the scale bar with the coded markers.

And I'll drag and drop them
into the user interface. I'm importing only these six
images with the scale bar to speed up the detect
markers algorithm. With all of the images, it would
take significantly more time. If I expand the images node in
D1DS and select one of the images,

for example, this one, a new 2D view
of the image will be displayed. Here I can zoom in with
the mouse scroll wheel. And here we can see the markers. They serve to automate the
process of placing control points.

I manually placed control
points in the tutorial about creating a winter environment
using Reality Capture, Quixel Mixer, and Twinmotion. The link is in the description. The detect markers algorithm
will place control points on these markers. And then I will be able to
define a distance constraint.

The Detect Markers tool is located
in the Alignment tab, right here. It can be used not just
for detecting the markers, but it can also generate them. In the generator, I can specify
how many markers I want, how many should be per page, and
the paper size, which I left at A4.

In the end, I only used two markers. The marker type can be
selected at the top. I use the first default option,
circular, single ring, 12-bit. To detect the printed markers, the
same marker type must be selected.

Here I can set the required
number of measurements. I usually increase this value a
bit to prevent false positives. I usually use three, but it
wouldn't matter for these six images because I don't expect
any false positives.

Finally, I can select the image layer that is supposed to be used for detecting the markers. Right now, I only have a single image layer, which is the geometry layer. To run the algorithm, I'll press "Detect". The Detect Markers algorithm did its job.

When I expand the control points node in D1DS, we see two new control points, both of which were detected on all six images. They are also visible directly in the images displayed in the 2D view. Now I want to tell RC that the distance between these two markers

is 60 centimeters. I'll use the distance definitions from the Workflow tab to import a text file with the definition. I will bring in my Explorer window and open the distance definition. The file contains only one distance definition,

but there could be multiple. It has four columns, and the first column is the name of the distance. So, in this case, it's distance 1. The second and third columns are the names of the control points that will define the distance.

They have to match the names in RC, and finally, the actual distance in the current system units. I'm using the default local Euclidean coordinate system with meters as the units. So this distance is 60 centimeters. Now I can close it and get this window out of the way.

I'll click on distance definitions, select the file, and click "Open". In the import dialog, I have to match the file format and value separator with the file containing the distance definition. These are the defaults. And I created the file with these default settings in mine.

So it should be correct. The file format is the name of the distance, point A, point B, and the distance itself. That is correct. The value separator is a space, which is also correct. There is no need to ignore the first line.

And the coordinate system is local Euclidean. In the Imported Data Accuracy, I can also edit the accuracy of the measurement. But I'll leave it at the default value of 1 millimeter. I used a ruler with millimeter divisions to measure the distance.

So 1 millimeter is appropriate. Now I can click on OK. When I expand the constraints node in D1DS, as we can see the distance 1. In the properties, we can see that the file was correctly imported.

We don't have to calculate the distance just yet because we don't have an existing alignment. Before the alignment, I'll import the rest of the images. Again, I'll bring up my Explorer window. This time I can select the entire folder with the images

and drag and drop it into the user interface. RealityCapture will recognize that the first six images are already imported, so there won't be any duplicates. And now I can just click on 'Align Images'. The alignment is finished.

In the 3D view, we can see the sparse scene reconstruction and the camera positions. In D1DS, we can see that RealityCapture created one large component that contains most of the cameras and a bunch of small components that contain from 2 to 10 cameras.

I shot way more images than I needed for a successful reconstruction, so a couple of online images shouldn't be a problem. I don't see any holes in the reconstruction, so the final mesh should be fine. If you want to know which cameras are not aligned in this large component,

ensure it is selected, switch from D1DS to 2DS, go to the Scene 2D View tab, and click on Show Unregistered. There are a bunch of detail shots and some images from the inside of the tower that were not aligned.

As I said, a couple of missing images shouldn't be a problem. There could be a couple of reasons why they are not aligned. There could be blurry images, not enough common features between neighboring images, or simply they were above the accuracy threshold

and RealityCapture decided not to use them. You may want to try aligning them. Sometimes a simple second alignment can do the trick, or you can use control points and realign with the control points. I won't bother with that this time, so the next step

is to set up the ground plane in the reconstruction region. I'll go to the Scene 3D Tool step and click on Set Ground Plane to activate the widget. I'll switch the layout to 1+1 to have a large 3D view.

I like to adjust the ground plane with the help of orthographic views. They can be toggled in the Scene 3D View tab right here. Or you can use the number pad. So I will do that. To enter the top view, you can press the number 2 on the number pad.

First, I'll rotate the scene so the front of the hunting tower faces the right side. I'm doing this so the exported model has the proper orientation in Unreal Engine. Besides rotating the scene, I'll try to place the tower in the middle

of the grid, something like this. Next, I will switch to one of the side views by pressing 5 on the number pad. In this view, I'll check if the tower is in the middle of the grid. And I will adjust the bottom of the scene so it sits on this black line.

OK, that should be good. To get back to the Perspective View, I will press number 0 on the number pad. Now I can deactivate the Set Ground Plane tool by clicking on it again. To reset the reconstruction region, I'll

click on the Set Reconstruction Region command. And, again, adjust it in the orthographic views; number 2 on the number pad to enter the top view. I'll use the box widget to adjust the size. If the widget is not visible, just click on the reconstruction region

with the left mouse button. I'll also switch to the Side View by pressing number 5 on the number pad, and I will adjust the top and the bottom. I'll press number 0 on the number pad to enter the Perspective View.

And that's it. To launch the reconstruction, I will go to the Mesh Model tab and click on 'Create Model On Normal Detail'. The reconstruction is now finished. I'll switch the 2DS back to 1DS. If I expand component number 10, we can see that the model number

Model 1 has around 139 million triangles. In the 3D View, we can see it in the form of a very dense point cloud. And the reason for this is that it has so many polygons. RealityCapture's real-time render limit is 40 million

to ensure a smooth experience. Before I do any texturing and simplification, first I want to do some cleanup. RealityCapture's meshing algorithm creates watertight meshes. In this case, it filled out the bottom with large triangles.

We call them marginal triangles. Right now they are not visible because of the point cloud rendering. To display them, I'll use the Clipping Box. I will go to the Scene 3D View tab, expand the Clipping Box,

and click on 'Create From Reconstruction region'. Now I'll enable editing and bring the top of the Clipping Box all the way down. I will also make it larger so it doesn't clip the marginal triangles. The mesh parts should start loading if the rendering

mode is set to solid. Now the large triangles are visible. I want to get rid of them. Now, there are a couple of ways I can do that. I could go to the Scene 3D Tools tab and use the Advanced Selection

tool to select the marginal triangles and filter them out like I did in the winter environment tutorial. But this time I will try something different. I will disable the tool, deselect the triangles, and use the Cut By Box tool instead.

First, I'll select the reconstruction region and slightly adjust it. I'll enter the Top View and make the reconstruction region slightly smaller, like this. Next, I'll go to the Side View by pressing the number 5 on the number

pad and adjust the bottom. I want to cut everything outside of this reconstruction region, so I'll use Cut Outer. Fill Cut Holes should be set to no because that's not what I want. Now, I will just click on Cut Out.

The Cut By Box is finished, so we have a second model. This time, it is without large triangles at the bottom. Now I can get rid of the Clipping Box, so I will go to the Scene 3D View tab and clear the Clipping Box.

Now I have a couple of options. I could simplify the model and texture it later. But I prefer to texture a clean, highest-resolution model I have with the fixed texel size and optimal texel size to get the best possible texture quality.

Then I'll use this high-poly version to create simplified versions and bake the textures. To texture it, I'll go to the Mesh Model tab and click on the texturing settings. In the unwrap parameters, I'll switch the maximal texture resolution

from 8K to 16K. Next, I'll set the style to Fix Texel size. I'll keep the texel size set to optimal, which is the smallest possible texel that provides the best detail. This way, I will get the best possible texture.

That's what I can get from the input images. I'll also increase the large triangle removal threshold to 100 to ensure that even larger triangles are properly unwrapped and textured. Now I can start texturing. The texturing is finished, so now the model has colors.

And we can see them in the 3D View. If I expand model number 2, we can see that the fixed texel size, with the optimal texel size, created a 516K texture, which is a lot. The next step is the simplification of this model.

In this step, I will create the final model that I will use in Unreal Engine 5. To do that, I will use the Simplify tool. The tool is located in the Scene 3D Tool step, right here.

I want to use the model with 10 million polygons in UE5, so I'll keep the type set to absolute. And I'll set the target triangle count to 10 million. In the winter environment tutorial, I used the Simplification tool

to also bake the color and normal map textures. This time I will keep the texture reproduction disabled because I want to show you another way that gives you more control over the unwrapping of the simplified model and also more control over the texture reprojection.

Now, everything is set, so I'll click on Simplify. The simplification is now finished. We have a new model, number 3, with 10 million polygons. Because the texture projection was disabled, it doesn't have textures or UVs. We'll do them manually.

The colors we see in the viewport are only vertex colors. First, I'll create the UVs with the Unwrap tool. I'll disable the Simplify tool and close the Texturing settings. The Unwrap tool is in the Mesh Model tab, right here.

For texturing the high-poly mesh, I used a fixed texel size that created a 516K texture. I could use the same settings, but that resolution isn't necessary. I'll use a 116K texture.

I'll leave the style set to the maximal texture count, and the maximal texture count set to 1. I'll also increase the large triangle removal threshold to 100 and click Unwrap. The unwrap is finished.

OK, now I'll use the Texture Reprojection tool to bake the texture. I'll disable the Unwrap tool and enable texture reprojection. The tool is located next to the Simplification tool in the Scene 3D Tools tab.

You might ask why I'm using this tool.  It takes significantly less time to bake the texture from the high-resolution source model than to texture the simplified model from scratch. I need to specify the source model; that's my high-resolution model

number 2. My result model is this model number 3. Color reproduction is already enabled. I can also enable normal map texture reprojection. I went from 139 million polygons to 10 million polygons.

So many lost details can be stored in the normal map texture. Now I'll press Preproject to start the process. The texture reproduction is now finished. Model number 3 has a 116K color texture and a 116K normal map texture.

This is my final model and texture. And now I will export the model as an FBX. I'll click on the RC icon and click on Export. I already have FBX in my recently used formats. I'll pick where I want to save it.

That will be my export folder, and I will simply call it Tower, and click on Save. In the export dialog, I'll enable Export Normals and disable Export Vertex Colors because I won't need them. The texture will be saved as a PNG, which is fine with me.

The coordinate system will be the grid plane. And then in the Export Transformation settings, I'll pick the Unreal Engine preset. This preset scales the mesh 100 times, and then I don't have to modify the scale during import to Unreal.

This preset also flips the Y channel of the normal map so it properly works in Unreal. Now I will click on OK. And that's it. I'm finished in Reality Capture. And now I will jump into Unreal Engine to import this model.

If you don't have Unreal Engine 5 already installed, go ahead and install it through the Epic Games Launcher. In the Marketplace, search for the medieval game environment and create a new project with it. Once it's downloaded, open the project with UE5.

Here I am in the Unreal Engine 5 editor with the medieval game environment already loaded. I have the main startup level loaded, and everything works. Straight out of the box, I'll just import the mesh from RealityCapture, make a basic setup, and place

it in the level. Now, first I recommend you go and explore this environment. It is also a fully playable experience, so definitely try it out. There's a lot of things going on in the 3D viewport,

so I will press G on the keyboard to enter the Game View mode. That will get rid of everything. The first thing I need to do is import my mesh from RealityCapture. I will open the Content Browser, and I will click on Dock And Layout.

In the Content Browser, I want to make these icons smaller, so I will press Control and use the mouse scroll wheel. In the Content folder, I'll add a new folder and call it RealityCapture. I'll press Enter to confirm the name, and I'll double

click it to open the folder. Now I'll click on Import. In the Explorer, I will select my mesh from RealityCapture and click on Open. In the import options, I am using the default settings. Just make sure that Build Nanite is checked.

Unreal will also import my textures, and it will also create a material for the model. Now I can just click on Import. And now, based on the size of your mesh, it may take a couple of minutes to import the asset into Unreal.

The assets are loaded, so we can see the mesh, the material, and the textures. You can rename the assets if you want, but I'll keep the original names. This star icon on an asset means it hasn't been saved yet.

So I will click on Save All and wait. The assets are saved. Virtual texture support is enabled for this project. And because of the 16K resolution of the textures, virtual texture streaming was automatically enabled for them. It can be disabled in the texture details.

The auto virtual texturing setting size can be changed in the Project settings. Now, I will open the material that was created for me because I want to edit it. I want more control over some settings; mainly, I want to control the brightness of the color texture,

and I also want to control the roughness of the material. I'll use a Multiply node and a scalar parameter to control the brightness of the texture. I'll press M on the keyboard and left-click in the node graph to create the multiply node.

Now, I also need a scalar parameter to control the brightness. So I will press S on the keyboard and again click in the node graph. Right away, I will rename this parameter to "Brightness." I will connect brightness to the A input

and I will connect the texture to the B input. The order doesn't matter because it's multiplication. I will connect the output of the multiply node to the base color of the material. Right now, the brightness is set to 0.

And when you multiply anything by 0, the result is 0. So in my case, that returns a black color for the material. That's not exactly what I want, so I will set the default value to 1.

That means the brightness won't change. To control the roughness, I will again create another scalar parameter. So I will press S on the keyboard and click in the node graph. This time I'll rename it to roughness.

And I'll connect it to the roughness of the material. I will set the default value to 0.5. That's a good starting point, but I will tweak it later in the material instance. So now I can save and close it.

From this material, I will create a material instance. The reason why I'm doing this is to have control over the brightness and roughness of the material in real time. Now when I open it, I can enable the options

for editing the brightness and roughness. I won't make any changes just yet. First, I'll apply the material instance to the mesh. Let's minimize this for now. I'll open the mesh. Currently, the mesh is using the original material.

So to apply the material instance, I will just drag and drop the material instance into the material slot. Now I can save the mesh. The mesh is saved, so now I can close it. For now, I will only minimize the material instance.

Now I will place the tower into the level. And I think that somewhere around here will be a good place for the tower, so I will move in closer. I will drag and drop the tower into the level like this.

I will switch to rotation. And I will rotate the tower so it faces the field. Now I will switch back to Translate Objects and I will move it back like this. I also want to hide the original scan ground, so I will move it down.

I think that's good enough. Now, right away, we can see that the tower is too bright. So I'm going to edit the material instance. I will switch from lit mode to unlit. And now we can see that the brightness is even more obvious.

I will open the material instance and set the brightness to around 0.4. And now I think that the color of the tower blends into the environment more naturally. Now I'm going to switch the unlit mode to a Buffer Visualization.

Roughness. We can see that right now it also stands out. White means 100% roughness, and black means no roughness. I will increase the roughness to, let's say, 0.9. I think that this is more appropriate for the tower.

Now I will switch back to the lit mode. And I think that the tower blends into the environment more naturally. You definitely can spend way more time on this, making a more complex material, and tweaking everything. But I think it's time for me to play the experience.

So I will save the material instance and close it. Now I can press Play. I hope you enjoyed watching this tutorial as I did making it. We covered every single step of making and bringing the asset from reality capture to Unreal Engine 5.

You should be able to turn your own scans into UE5 assets with ease. If you have any thoughts or questions, feel free to drop them in the comments of this video. See you in the next one.

RealityCapture to Unreal Engine 5

RealityCapture to Unreal Engine 5

Scripts