RealityServer Features

Picking

The default render loop handler provides the ability to perform a pick in the image much like the scene_pick command. Performing a pick will return details about the front most object found under the pixel or area covered by the pick.

The render loop sample application contains an example of how to pick an object in a render as part of the outline functionality.

Triggering requests and receiving results

Similar to auto exposure picking is an asynchronous process. Consequently it uses the same request/response procedure as auto exposure to trigger picks and receive their responses except it uses the pick and pick_response attributes. Please see the Auto Exposure Triggering documentaion for more details.

Picking Trigger Camera Attributes
Attribute Name Attribute Value Attribute Type Description
pick_request none Sint32 Change this to a value other than 0 to set off a pick.
pick_response none Sint32 When this attribute has the same value as the request the pick is complete.

Example: Trigger a pick 

let pick_request = 0;

pick_request++;
send_realityserver_command(
    {
        "method": "element_set_attributes",
        "params": {
            "element_name": "my_camera",
            "attributes": {
                "pick_request" : {
                    "value": pick_request,
                    "type": "Sint32"
                },
                "pick_position" : {
                    "value": {x: 150, y: 150},
                    "type": "Float32<2>"
                }
            },
            "create": true
        }
    }
);

Example: Poll for the result 

let pick_response = send_realityserver_command({
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_response"
            }
        }
    );
while (pick_response !== pick_request) {
    sleep(5); // wait 5 seconds then poll again
    pick_response = send_realityserver_command({
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_response"
            }
        }
    );
}
// pick is complete, the results can now be fetched off the camera.
Note: In the above the function send_realityserver_command is a synchronous function that sends the provided command or array of commands and returns it's result or undefined on error.

Pick settings

Picking is controlled via various attributes on the scene camera. pick_position is the point to pick and setting pick_size allows an area to be picked rather than a point.

Pick Camera Setting Attributes
Attribute Name Attribute Type Description
pick_position Float32<2> The pixel coordinate in the image to pick. Note that the y coordinate is bottom up.
pick_size Float32<2> If set then an area of this size centered around pick_position is picked rather than a point.

Pick results

Once complete the following attributes are set on the scene camera to provide the results. If no error occurred they will contain information about the frontmost object found.

Pick Camera Result Attributes
Attribute Name Attribute Type Description
pick_result_error String If an error occurred this attribute will contain an error message, otherwise it will be an empty string.
pick_result_position Float32<3> The world space coordinate of the pick result.
pick_result_object_name String The name of the object picked.
pick_result_path String[] Contains the path through the scene DAG to get to the instance of the picked object.

Example: Pick an object and show results 

pick_request++;
const pick_pos = { x: 150, y: 100 };
send_realityserver_command(
    {
        "method": "element_set_attributes",
        "params": {
            "element_name": "my_camera",
            "attributes": {
                "pick_request" : {
                    "value": pick_request,
                    "type": "Sint32"
                },
                "pick_position" : {
                    "value": pick_pos,
                    "type": "Float32<2>"
                }
            },
            "create": true
        }
    }
);

let pick_response = send_realityserver_command({
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_response"
            }
        }
    );
while (pick_response !== pick_request) {
    sleep(5); // wait 5 seconds then poll again
    pick_response = send_realityserver_command({
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_response"
            }
        }
    );
}
// pick is complete, the results can now be fetched off the camera.
const [err,pos,object_name,path] = send_realityserver_command(
    [
        {
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_result_error"
            }
        },
        {
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_result_position"
            }
        },
        {
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_result_object_name"
            }
        },
        {
            "method": "element_get_attribute",
            "params": {
                "element_name": "my_camera",
                "attribute_name: "pick_result_path"
            }
        }
    ]
)
if (err.length) {
    alert('Error picking at ' + pick_pos.x + ',' + pick_pos.y + ':  ' + err)
} else {
    path.push(object_name);
    alert('Picked ' + path.join('|') + ' at location ' + pos.x + ',' + pox.y + ',' + pos.z + ' (' + pick_pos.x + ',' + pick_pos.y + ')');
}