みらいテックラボ

音声・画像認識や機械学習など, 管理人が興味のある技術の紹介や実際にトライしてみた様子などメモしていく.

Jetson NanoにCSI-2カメラを繋いでみたー(1)

普段, Jetson Nanoでカメラを使う際にはUSBタイプを使うのだが, 今回初めてMIPI CSI-2タイプのカメラを試してみた.
今回使ったカメラはこれ.
・Jetson Nano Camera IMX219PQ 8M Color CMOS Sensor IR-Filter Wide Angle Fished eye Len FOV158 degree

OpenCVでこれを使おうとすると, USBタイプのようにはいかなかったので, メモとしてまとめておく.

[Jetson Nano]

f:id:moonlight-aska:20210809173401j:plain:w400


関連記事:


0. まずはどうなる?
カメラ画像をキャプチャするのに使用するOpenCVのVideoCapture()をUSBカメラの時と同じように使用するとどうなるか?

[ソース; camera_test.py]

import sys
import cv2

cap = cv2.VideoCapture(0)
if not cap.isOpened():
    print('Can not open camera.')
    sys.exit()

while True:
    ret, frame = cap.read()
    if not ret:
        break
    frame = cv2.resize(frame, (640, 360))
    cv2.imshow('Viewer', frame)
    if cv2.waitKey(30) == 27:
        break

cv2.destroyAllWindows()
cap.release()

[実行結果]

$ python3 camera_test.py
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (1757) handleMessage OpenCV | GStreamer warning: Embedded video playback halted; module v4l2src0 reported: Internal data stream error.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (886) open OpenCV | GStreamer warning: unable to start pipeline
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (480) isPipelinePlaying OpenCV | GStreamer warning: GStreamer: pipeline have not been created
Gtk-Message: 17:48:31.473: Failed to load module "canberra-gtk-module"

f:id:moonlight-aska:20210809095949p:plain:w400

バイスのオープンエラーとはならないが, 正しく表示されない.
"ls /dev/video*"でカメラデバイスを確認したとき, usbカメラと同じように"/dev/video0"が確認できたので, usbカメラと同様に扱えると思ったのだが, どうも違うようだ.


1. カメラ情報の確認
v4l-utilsを使って, カメラ情報を確認してみる.

$ sudo apt install v4l-utils

$ v4l2-ctl --list-devices
vi-output, imx219 6-0010 (platform:54080000.vi:0):
             /dev/video0

$ v4l2-ctl -d /dev/video0 --all
Driver Info (not using libv4l2):
	Driver name   : tegra-video
	Card type     : vi-output, imx219 6-0010
	Bus info      : platform:54080000.vi:0
	Driver version: 4.9.140
	Capabilities  : 0x84200001
		Video Capture
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps   : 0x04200001
		Video Capture
		Streaming
		Extended Pix Format
Priority: 2
Video input : 0 (Camera 0: no power)
Format Video Capture:
	Width/Height      : 3264/2464
	Pixel Format      : 'YUYV'
	Field             : None
	Bytes per Line    : 6528
	Size Image        : 16084992
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                    sensor_mode 0x009a2008 (int64)  : min=0 max=0 step=0 default=0 value=0 flags=slider
                           gain 0x009a2009 (int64)  : min=0 max=0 step=0 default=0 value=16 flags=slider
                       exposure 0x009a200a (int64)  : min=0 max=0 step=0 default=0 value=13 flags=slider
                     frame_rate 0x009a200b (int64)  : min=0 max=0 step=0 default=0 value=2000000 flags=slider
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
               write_isp_format 0x009a2068 (bool)   : default=0 value=0
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=0 step=0 default=0 flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=6 flags=read-only

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'RG10'
	Name        : 10-bit Bayer RGRG/GBGB
		Size: Discrete 3264x2464
			Interval: Discrete 0.048s (21.000 fps)
		Size: Discrete 3264x1848
			Interval: Discrete 0.036s (28.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.017s (60.000 fps)


2. GStreamerの利用[1]
MIPI CSI-2カメラについてネットを調べていると, JetsonHasksNanoでサンプルコードを発見!!
GStreamerのpipelineを利用するようだ.
これを参考に, OpenCVのVideoCaputer()で映像を表示することができた.

[コード]

import sys
import cv2

# GStreamer pipeline
def gstreamer_pipeline(
    capture_width=1280,
    capture_height=720,
    display_width=1280,
    display_height=720,
    framerate=60,
    flip_method=0,
):
    return (
        "nvarguscamerasrc ! "
        "video/x-raw(memory:NVMM), "
        "width=(int)%d, height=(int)%d, "
        "format=(string)NV12, framerate=(fraction)%d/1 ! "
        "nvvidconv flip-method=%d ! "
        "video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
        "videoconvert ! "
        "video/x-raw, format=(string)BGR ! appsink"
        % (
            capture_width,
            capture_height,
            framerate,
            flip_method,
            display_width,
            display_height,
        )
    )

cap = cv2.VideoCapture(gstreamer_pipeline(display_width=640, display_height=360), cv2.CAP_GSTREAMER)
if not cap.isOpened():
    print('Can not open camera.')
    sys.exit()

while True:
    ret, frame = cap.read()
    if not ret:
        break
    cv2.imshow('Viewer', frame)
    if cv2.waitKey(30) == 27:
        break

cv2.destroyAllWindows()
cap.release()

[実行結果]

$ python3 camera_test.py
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1640 x 1232 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 5 
   Output Stream W = 1280 H = 720 
   seconds to Run    = 0 
   Frame Rate = 120.000005 
GST_ARGUS: PowerService: requested_clock_Hz=12096000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
[ WARN:0] global /home/nvidia/host/build_opencv/nv_opencv/modules/videoio/src/cap_gstreamer.cpp (933) open OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
Gtk-Message: 00:12:37.718: Failed to load module "canberra-gtk-module"

f:id:moonlight-aska:20210809173823p:plain:w400

Warningが出ているが, 一応カメラ画像をキャプチャし, 表示することができた.
ただ, 周囲が少し赤みがかるなど, 画質があまりよくない.

このカメラはシャッタースピードや絞りの設定などが行えるのか, 少し調べてみようと思う.

----
参照URL:
[1] JetsonHacksNano/CSI-Camera