First Azure Kinect App–VIDEO- Getting Depth and Color camera

Hello!

This article if part of a serie of “How to” create Apps with Azure Kinect DK. Yesterday I posted the first Part of this serie. Including:

Connect and open our Device
Configure our camera
Start our camera
Stop our camera
Dispose our Device

Today I am excited to dig in the theory of the configuration of our camera. As you saw in first post we had the following configuration.

APPLICATION WITH C API

k4a_device_configuration_t config = K4A_DEVICE_CONFIG_INIT_DISABLE_ALL;
config.camera_fps = K4A_FRAMES_PER_SECOND_30;
config.color_format = K4A_IMAGE_FORMAT_COLOR_MJPG;
config.color_resolution = K4A_COLOR_RESOLUTION_2160P;
config.depth_mode = K4A_DEPTH_MODE_NFOV_UNBINNED;

APPLICATION WITH K4A NUGET PACKAGE

DeviceConfiguration config = DeviceConfiguration.DisableAll;

config.CameraFps = FrameRate.Thirty;

config.ColorFormat = ImageFormat.ColorMjpg;

config.ColorResolution = ColorResolution.R2160p;

config.DepthMode = DepthMode.NarrowViewUnbinned;

But the thing is… what did we configurate in here?

“Camera Frames Per Second.”

FPS is used to measure frame rate – the number of consecutive full-screen images that are displayed each second. On average, the human eye can process 12 separate images per second. This means a frame rate of 12 FPS can display motion, but will appear choppy. Once the frame rate exceeds 12 FPS, the frames appear less discrete and start to blur together. A frame rate of 24 FPS is commonly used for film since it creates a smooth appearance. Many video cameras record in 30 or 60 FPS, which provides even smoother motion.

In our Azure Kinect Sensor SDK we have a Color and Depth Sensor Frame Rate of: 5fps, 15fps and 30fps.

C API – K4A_FPS_T

Enumerator

K4A_FRAMES_PER_SECOND_5 | 5 FPS

K4A_FRAMES_PER_SECOND_15 | 15 FPS

K4A_FRAMES_PER_SECOND_30 | 30 FPS ///Smoother appearance

K4A NUGET PACKAGE

Enumerator

FrameRate.Five;

FrameRate.Fifteen;

FrameRate.Thirty;

“Image Format Type”

C API – K4A_FPS_T

K4A_IMAGE_FORMAT_COLOR_MJPG

Each image is encoded as a JPEG and can be decoded by JPEG decoder.

K4A_IMAGE_FORMAT_COLOR_NV12

Each image separates the luminance and chroma data, the luminance is at the beginning of the buffer and the chroma lines follow immediately after.

K4A_IMAGE_FORMAT_COLOR_YUY2

Each image stores chroma and luminance data in interleaved pixels.

K4A_IMAGE_FORMAT_COLOR_BGRA32

Each pixel of BGRA32 data is four bytes. The first three bytes represent Blue, Green, and Red data. The fourth byte is the alpha channel and is unused in the Azure Kinect APIs.

K4A_IMAGE_FORMAT_DEPTH16

Each pixel of DEPTH16 data is two bytes of little endian unsigned depth data. The unit of the data is in millimeters from the origin of the camera.

K4A_IMAGE_FORMAT_IR16

Each pixel of IR16 data is two bytes of little endian unsigned depth data. The value of the data represents brightness. This format represents infrared light and is captured by the depth camera.

K4A NUGET PACKAGE

ImageFormat.ColorMjpg;

ImageFormat.ColorNV12;

ImageFormat.ColorYUY2

ImageFormat.ColorBgra32;

ImageFormat.Depth16;

ImageFormat.IR16;

“Color Sensor Resolution”

C API – K4A_FPS_T

K4A_COLOR_RESOLUTION_OFF

Color camera will be turned off with this setting.

K4A_COLOR_RESOLUTION_720P

1280 * 720 16:9

K4A_COLOR_RESOLUTION_1080P

1920 * 1080 16:9

K4A_COLOR_RESOLUTION_1440P

2560 * 1440 16:9

K4A_COLOR_RESOLUTION_1536P

2048 * 1536 4:3

K4A_COLOR_RESOLUTION_2160P

3840 * 2160 16:9

K4A_COLOR_RESOLUTION_3072P

4096 * 3072 4:3

K4A NUGET PACKAGE

ColorResolution.Off;

ColorResolution.R720p;

ColorResolution.R1080p;

ColorResolution.R1440p;

ColorResolution.R1536p;

ColorResolution.R2160p;

ColorResolution.R3072p

”Depth sensor capture modes”

Read this article to understand the Field of View.

To resume: More Narrow its less Wide view. Binned modes reduce capture camera resolution.

K4A_DEPTH_MODE_OFF

Depth sensor will be turned off.

K4A_DEPTH_MODE_NFOV_2X2BINNED

320px x 288px Depth. Passive IR with the same values.

K4A_DEPTH_MODE_NFOV_UNBINNED

640px x 576px Depth. Passive IR same.

K4A_DEPTH_MODE_WFOV_2X2BINNED

512px x 512px Depth. Passive IR same.

K4A_DEPTH_MODE_WFOV_UNBINNED

1024px x 1024px Depth. Passive IR same.

K4A_DEPTH_MODE_PASSIVE_IR

Only Passive IR 1024px x 1024px.

DepthMode.Off;

DepthMode.NarrowView2x2Binned;

DepthMode.NarrowViewUnbinned;

DepthMode.WideView2x2Binned;

DepthMode.WideViewUnbinned;

DepthMode.PassiveIR;

Now that we understand the different parameters we can manage with the sensor lets understand how to provide an image with this information to our App. Each captured image contains a depth image, an IR image, a color image, or a combination of images.

In code behind there is a timer that overwrites the WriteableBitmap ten times per second with a random pixel color value. Note that you have to allow unsafe code in the Visual Studio Project Properties (in the build tab).