.. _rgbdgrabber: RGBD Grabber ===================== The RGB-D Grabbing and streaming is done using a combination of OpenCV and Gstreamer in the Person Tracking components. This captures images from an ASUS Xtion Pro Live USB camera mounted at the apartment's ceiling. For each camera two video streams and an audio stream are created and provided via the rtp protocol. Related resources ----------------- Component repository: - Browse component repository (master): `lsp-csra.person-tracking `_. - ``git clone https://projects.cit-ec.uni-bielefeld.de/git/lsp-csra.person-tracking.git`` Depth image encoding -------------------- Depth images cannot be directly encoded in h264 so some transformations need to be done by the person-tracking before the images are passed into the gstreamer pipeline. The distances in millimetres are cut off at 4096mm and scaled by 1/16 to fit into an 8bit image. In c++ code: .. code-block:: cpp cv::Mat input_depth = grab_depth_image_from_camera(); cv::Mat transformed_depth; // scaling distance from [0,4096) mm to [0,256) (factor = 1/16) everything >= 4096 is cut off imput_depth.mat.convertTo(transformed_depth, CV_8UC1, 1.0 / 16); // mask all cut-off values to zero. transformed_depth.setTo(0,transformed_depth == 255); cv_video_writer.write(transformed_depth); Cameras and hosts ------------- A RGB-D device (ASUS Xtion Pro Live) is connected to one camera host (Intel NUC barebone) at a time. Below the list of installed camera hosts: =============== ==== =========== ========= ========== ========== ========== Camera location Host Stat./Info UDP Group Color Port Depth Port Audio Port =============== ==== =========== ========= ========== ========== ========== Hallway (CAM1) dme available 224.2.1.1 5200 5250 5300 Hallway (CAM2) iom available 224.2.1.1 5201 5251 5301 Kitchen (CAM1) mpc available 224.2.1.1 5202 5252 5302 Kitchen (CAM2) lxr available 224.2.1.1 5203 5253 5303 Living (CAM1) bfe available 224.2.1.1 5204 5254 5304 Living (CAM2) bos available 224.2.1.1 5205 5255 5305 Living (CAM3) adl available 224.2.1.1 5206 5256 5306 Living (CAM4) bkk available 224.2.1.1 5207 5257 5307 Sports (CAM1) dxb !disabled! Sports (CAM2) atl !disabled! Sports (CAM3) pug !disabled! Sports (CAM4) bcn !disabled! =============== ==== =========== ========= ========== ========== ========== Accessing the streams --------------------- A single rgb stream can be accessed via gstreamer with the following pipeline: .. code-block:: bash gst-launch-1.0 \ udpsrc do-timestamp=true address=224.2.1.1 port=5200 \ ! application/x-rtp ! queue ! rtph264depay ! h264parse ! avdec_h264 \ ! videoconvert ! queue ! xvimagesink sync=false The same pipeline can be used with OpenCV with Gstreamer support. Just drop the `gst-launch-1.0`, switch `xvimagesink sync=false` for 'appsink' and pass the whole thing as parameter to a cv::CaptureDevice. Visualisations of the depth and color streams can be started with the scripts: .. code-block:: bash # shows color video streams from ceiling and webcams in a matrix gst-show-csra.sh -m -r # shows depth video streams from ceiling gst-show-csra-depthcams.sh Examples -------------- Starting the Person-Tracking & Grabber component: This example is the configuration used in the csra. The images can be flipped and get an overlay with the current time. The encoding is done with hardware support if available. As a fall back, software encoding is used. Colour and depth streams are encoded as h264 video streams. Audio is grabbed too and encoded in ac3. The streams are published separately via rtp using udp. .. code-block:: bash PORT=5000 FLIP=0 CLOCKOVERLAY=" ! clockoverlay \ time-format=\"%Y-%m-%d %H:%M:%S %Z\" \ halignment=left valignment=top \ xpad=0 ypad=0 auto-resize=true \ shading-value=255 shaded-background=true \ font-desc=r\"Sans,\ 8\" draw-shadow=false \ draw-outline=false " # need the right display for hardware encoding export DISPLAY=:0 # test if vaapi is supported if gst-launch-1.0 videotestsrc num-buffers=1 ! vaapipostproc ! vaapih264enc ! fakesink; then CONVERT=" vaapipostproc ! vaapih264enc cabac=true tune=high-compression " else CONVERT=" x264enc speed-preset=veryfast tune=zerolatency " fi # find PrimeSense audio device AUDIO_DEVICE=$(arecord -l | grep "PrimeSense" | sed -r "s/^card[ ]([0-9]).*/\1/g") AUDIO="hw:${AUDIO_DEVICE},0" # set audio capture volume amixer -c$AUDIO_DEVICE set 'Mic',0 50% amixer -c$AUDIO_DEVICE set 'Mic',1 50% export GST_DEBUG=4 rsb_monitor rsb-person-tracking -o $PH_SCOPE -c $CAM_ID -f Home \ -r "appsrc ! queue ! videoconvert ! videoflip method=$FLIP $CLOCKOVERLAY \ ! $CONVERT ! queue ! h264parse ! rtph264pay \ ! udpsink host=224.2.1.1 port=$PORT auto-multicast=true \ alsasrc device=\"$AUDIO\" ! audioconvert ! avenc_ac3 ! ac3parse \ ! rtpac3pay ! udpsink host=224.2.1.1 port=$(($PORT+100)) auto-multicast=true" \ -D "appsrc ! queue ! videoconvert ! videoflip method=$FLIP $CLOCKOVERLAY \ ! $CONVERT ! queue ! h264parse ! rtph264pay \ ! udpsink host=224.2.1.1 port=$(($PORT + 50)) auto-multicast=true" \ rsb-depth-sensors-grabber -s "PARAMETER"