360 Video Conferencing with the RICOH THETA S

video

#1

You can live stream 360 video between two computers using a browser. The video stream has full 360 degree navigation, enabling 360 video conferencing between two people.

The magic happens with WebRTC. A functional sample application is available with source code. WebRTC requires an external server for signaling. In this guide, we use the RICOH Cloud API server for signaling. Free accounts are available. The video data is sent between the two browsers and does not travel to the signaling server. In addition to a login to the signaling server, you will need API keys. These are also freely available. Refer to the link above for more information.

Download live streaming code from GitHub.

Note that if you don’t have git installed and don’t know how to install it from the command line, you can download the GitHub Desktop.

Follow the steps and get going.

$ git clone https://github.com/ricohapi/video-streaming-sample-app.git
$ cd video-streaming-sample-app/samples

Remain in the directory samples

$ cp config_template.js config.js

Put your credentials into config.js The sample application uses the RICOH Cloud as the signaling server. You may be able to modify the source code of the sample application and build your own signaling server. For simplicity, this example focuses on using the free RICOH Cloud as the signaling server.

The credentials you get from the RICOH Cloud API server look like this.

If you don’t have your free API key credentials for signaling, read this RICOH Cloud API signup guide.

$ npm install

When you run npm install, there will be warnings. Ignore them. I’ve tested Mac and Windows 10 with nodejs downloaded from the main nodejs.org site and the sample application ran with no problem. I’ve also tested it on Ubuntu 16.04 with both the node installed with apt install nodejs as well as the more recent version of node installed from the nodejs.org site.

$ npm run build

Go into the directory twoway

$ pwd
.../video-streaming-sample-app/samples/twoway

If you try running

$ npm start

and you see the error message about gulp: not found, then adjust your PATH for gulp

gulp error message

    $ npm start

    > @ start /home/craig/Documents/Accounts/ricoh/cloudApi/video-streaming-sample-app/samples/twoway
    > gulp server

    sh: 1: gulp: not found

    npm ERR! Linux 4.4.0-28-generic
    npm ERR! argv "/usr/bin/nodejs" "/usr/bin/npm" "start"
    npm ERR! node v4.2.6
    npm ERR! npm  v3.5.2
    npm ERR! file sh
    npm ERR! code ELIFECYCLE
    npm ERR! errno ENOENT
    npm ERR! syscall spawn
    npm ERR! @ start: `gulp server`
    npm ERR! spawn ENOENT
    npm ERR! 
    npm ERR! Failed at the @ start script 'gulp server'.
    npm ERR! Make sure you have the latest version of node.js and npm installed.
    npm ERR! If you do, this is most likely a problem with the  package,
    npm ERR! not with npm itself.
    npm ERR! Tell the author that this fails on your system:
    npm ERR!     gulp server
    npm ERR! You can get information on how to open an issue for this project with:
    npm ERR!     npm bugs 
    npm ERR! Or if that isn't available, you can get their info via:
    npm ERR!     npm owner ls 
    npm ERR! There is likely additional logging output above.

adjusting path to find gulp

If your system can’t find gulp, then do this:

$ npm bin

It will return your path, probably something like this:

/home/ricoh/Development/livestream/video-streaming-sample-app/samples/node_modules/.bin

Add the PATH for gulp.

On Linux systems, this will look similar to this:

 export PATH=$PATH: /home/ricoh/Development/livestream/video-streaming-sample-app/samples/node_modules/.bin

I’ve tested this on Windows 10, Linux Ubuntu 16.04, and Mac. In all cases, I needed to specify the PATH.

After the PATH environmental variable is set, then run npm start again.

$ npm start

> @ start /home/craig/Documents/Accounts/ricoh/cloudApi/video-streaming-sample-app/samples/twoway
> gulp server

(node:24274) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.
[15:51:30] Using gulpfile ~/Documents/Accounts/ricoh/cloudApi/video-streaming-sample-app/samples/twoway/gulpfile.js
[15:51:30] Starting 'server'...
[15:51:30] Webserver started at http://localhost:8034
[15:51:30] Finished 'server' after 9.61 ms

It will automatically open up a browser. RICOH recommends Google Chrome 51 or
newer. I also used Mozilla Firefox 47, which is the recommended version.

Select RICOH THETA S as the device.

Log in with your modified RICOH username and password.

If you don’t have a RICOH User account, refer to this guide.

To set up a peer-to-peer connection, use a Ricoh user ID with a plus sign and
string after it. For example, craig@oppkey.com+01 or craig@oppkey.com+theta.

In this example, I am using Chrome in two different tabs on the same computer. In the screenshot at the top of this section, the THETA is connected to one USB port of my Windows computer and another webcam (not a THETA) is
connected to another USB port on the same computer. The only reason I am only using one THETA is because I only have one THETA at my house.

I am using the THETA S driver, not UVC Blender. The preview window in the right of the browser is in dual fisheye.

Press the THETA View Mode green button to convert the incoming dual-fisheye video stream into a equirectangular view with the JavaScript in your browser.

So cool! The view has full navigation.

Streaming 360 Video Directly Between Two computers

To stream 360 between two computers on the same network, run the samples code on each computer. Open up a Chrome browser (Firefox didn’t work in my tests) on each computer. The sample application will look for a webcam
on each computer. If you only have one THETA, use your standard webcam, such as the one built into your laptop.

In my test network, I have the THETA connected to my Linux computer with a USB cable. My Linux computer is connected to my home router with WiFi on a private IP address in subnet 192.168.1.0. The router is connected to the Internet. Remember, Firefox on Linux did not work for me, only Chrome.

I have a second Windows 10 computer on the same subnet connected with Ethernet.

Here’s how the network looks.

Here’s what your screen may look like.

I’ve had problems connecting from Linux to Windows with the default firewall settings on Windows. If you have problems, turn off Windows firewall. You only need to connect from Computer A to Computer B, this is enough to set up a video streaming connection from each computer. Similar to using Skype, only one person needs to initiate the connection and then two-way video conferencing is set up.

Here’s a diagram of the streaming functionality of the application.

Here’s a diagram with some of the technology used.

Using WebRTC Between Two Computers on Separate Networks

To view the two 360 streams between two different computers on two separate networks (example: San Francisco and Palo Alto), you’ll need to assign each computer a static IP address on the Internet (not NAT on an internal network) or use NAT traversal.

I have not tried NAT Traversal yet.

You can check out two untested services for STUN / TURN for NAT traversal here:

Article on WebRTC

We have a full slide deck about the RICOH Cloud API available.

THETA Developer Ecosystem Registration


360 Live Streaming Guide for RICOH THETA
RICOH Demos 360 Large-scale Video Conferencing Cloud Platform
Contest Participant Profile: JavaScript and RICOH THETA Development
Mapping THETA S dual-fisheye live streaming to a spherical surface in the browser
Theta Refuses to live stream via USB - Error "Theta UVC Blender Status:0x800705AA"
RICOH Cloud API Beta Access Ends March 31st, 2017
Top 6 RICOH THETA Community Posts in 2016
Mapping THETA S dual-fisheye live streaming to a spherical surface in the browser
#2

Hi Criag,
Thanks for these instructions. I am waiting for my API access approval and will try this out. I have a question, do you know anything about the “Theta View Mode”? My goal is to develop an app and I was assuming I would have to use the UVC Blender, but it introduces a lot of latency (3-4 seconds). If I just do dual fish eye the latency is much lower. If there’s a way to render it in rectilinear, which it seems that this is what the Theta View Mode does, that would be amazing. Is that part of the SDK?


#3

https://github.com/ricohapi/video-streaming-sample-app/blob/master/samples/thetaview.js

Do this

$ git clone https://github.com/ricohapi/video-streaming-sample-app.git

or fork it.

There’s a small unequal gap in the stitching. @jcasman is supposed to publish a video of the demo. Hey, where is that video??? :slight_smile: The stitching quality may not be as good as YouTube, but it may be lower latency.


#4

Promise I’ll upload the video this week!


#5

@jcasman I’m looking forward to that masterpiece. hey, I don’t recall any delay between the movement and the video in the browser. Do you remember if there was a delay? Maybe test it in your office?

I know what @amiran is referring to about the delay on YouTube. It’s a bit of a problem for immersive experiences, but might be okay for most events, like a music concerts.


#6

The video quality maybe leaves something to be desired, but if there’s a delay, it’s very minor.


#7

@jcasman thanks for confirming. When you publish your video demo, people can see the quality and assess for themselves.


#8

Super quick intro video: Should give you a taste of what live streaming 360° video looks and feels like.


#9

Brilliant. Love it! Thanks for putting that video together.


#10

I’m working on a more elaborate one, but I figured short-and-sweet is fine in this case. Especially since the full blog post is right here for everyone to read.


#11

When are you going to publish the more elaborate video?


#12

I think I can have a convincing one finished early next week.


#13

Is there another video?


#14

I think an equirectangular view of the video conference might be better than navigation. ideally, the program would have multiple windows up at the same time to show multiple points of interest in the same stream. What do you think?

In this example, using Google Hangouts, the resolution looks good. There’s an API for the Google+ Hangout platform.

o


#15

Having issues getting this running. On the “npm start” command, I don’t get a gulp error, I get an error about a missing start script. Which previous step created this start script?

Here is the debug log:

0 info it worked if it ends with ok
1 verbose cli [ ‘C:\Program Files\nodejs\node.exe’,
1 verbose cli ‘C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js’,
1 verbose cli ‘start’ ]
2 info using npm@3.6.0
3 info using node@v5.7.0
4 verbose stack Error: missing script: start
4 verbose stack at run (C:\Program Files\nodejs\node_modules\npm\lib\run-script.js:147:19)
4 verbose stack at C:\Program Files\nodejs\node_modules\npm\lib\run-script.js:57:5
4 verbose stack at C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-json\read-json.js:345:5
4 verbose stack at checkBinReferences_ (C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-json\read-json.js:309:45)
4 verbose stack at final (C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-json\read-json.js:343:3)
4 verbose stack at then (C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-json\read-json.js:113:5)
4 verbose stack at ReadFileContext. (C:\Program Files\nodejs\node_modules\npm\node_modules\read-package-json\read-json.js:284:20)
4 verbose stack at ReadFileContext.callback (C:\Program Files\nodejs\node_modules\npm\node_modules\graceful-fs\graceful-fs.js:76:16)
4 verbose stack at FSReqWrap.readFileAfterOpen [as oncomplete] (fs.js:324:13)
5 verbose cwd C:\Users<redacted user name>\Documents\GitHub\video-streaming-sample-app\samples
6 error Windows_NT 6.1.7601
7 error argv “C:\Program Files\nodejs\node.exe” “C:\Program Files\nodejs\node_modules\npm\bin\npm-cli.js” "start"
8 error node v5.7.0
9 error npm v3.6.0
10 error missing script: start
11 error If you need help, you may report this error at:
11 error https://github.com/npm/npm/issues
12 verbose exit [ 1, true ]


#16

Fixed

There was an error in the documentation. You need to be in the sub-directory twoway when you run npm start

If you have an error message that gulp is not found, adjust your PATH to put gulp in the PATH, most likely INSTALL_DIRECTORY/video-streaming-sample-app/samples/node_modules/.bin

I fixed the documentation. Thank you for helping to debug it.


#17

Running “npm start” in the twoway directory successfully starts the server and opens a web browser to localhost:8034.

The web page renders blank. The source of the page matches ./index.html which is in the gulp.js file as a fallback.

The chrome console shows two uncaught syntax errors, one is an “Unexpected token <” in config.js line 1. In looking at the file in the chrome window, I don’t see the error, line 1 is “<!DOCTYPE html>”. Which “config.js” is this? Chrome resouces shows “config.js” as having the same contents as ./index.html.

The second error is “CONFIG not defined” in ricohapi-webrtc.js. Probably related to the first error.

Any more info on getting this running? I’m not familiar with a lot of this.

Bob


#18

In config.js, do you have the proper client_id and client_secret that you got from RICOH?

I’m going to post more details in a few minutes.


#19

Yes, I inserted the tokens that I got from signing up with Ricoh.


#20

Does that conifg.js file need to be moved to the “twoway” directory?