There isn’t a commercial solution for what I’m trying to do so I’ve created my own VR app using Unity.
I want the experience to be as close to real life as possible so I’m using Google StreetView data for this. Unfortunately this data is only captured every 4 meters or so but I want to be able to move smoothly through the world rather than in large steps. A single StreetView Panorama (Google’s name for the picture data at a given location) is recreated as a sphere around the viewer. I could move smoothly through that sphere from one Panorama to the next but the more I deviate from the center the more obvious it becomes that I’m just inside a sphere and not really surrounded by proper shapes. What I really needed was depth information to recreate the scene correctly. I was aware some simple depth was available from experience using StreetView: as I move the cursor over the scene a rectangle will appear in the correct size and shape depending on what I’m hovering over. After some searching I came across StreetView Explorer by Paul Wagener (which is based on the work by tbd)
This is a general purpose explorer allowing you to freely wander about but my quest is more specific. I’m only going to move between a start and end point and won’t deviate from that. This has advantages as we’ll see later. The code for this project was kindly available at Github so with that I could make this happen.
First my app starts by obtaining direction data from a start point to an end point using the url http://maps.googleapis.com/maps/api/directions/. The url also includes way points for specific places I want to visit along the way. Calling this will return me a huge list of latitude/longitude points in the order I will pass through. I then convert these into real world x/y meters (I just pretend the world is flat). Next I trace along those points and ask for the nearest Panorama as I go. I display this panorama until I realise I’m closer to a different one and then change. It’s a little more involved than that but that’s the basic principal.
The Panorama data is fairly simple to extract but using the StreetView Explorer gave me a huge jump start. The precious depth information is highly compressed and has been reduced to a list of about 100 flat planes. The result is… variable. For buildings it can look great given that they’re already flat. But for trees and other lumpy things it’s not so good. But most of the time things are far away enough that you don’t really care.
After recreating the scene the viewer is free to move about in it but the more the camera deviates from the center the more distorted some parts of the scene become depending on which plane an object has been plastered onto. In my first pass I simply moved the camera through the points returned from fetching the direction data but these points actually deviate quite a bit from where the Panoramas are captured. It worked but results looked distorted most of the time. To help correct this I now pass the camera through the Panorama origins so the direction data becomes more of a suggestion of where to get the data from.
The next piece of the puzzle was getting cadence data from my bike. I’m using a Bluetooth monitor and if I never have to use Bluetooth again I’ll be happy. It’s icky. Anyway some more research dug up this page which boosted me along but not all the way. It seems my device isn’t great at connecting and I spent a good deal of time trying to work that out. In the end I just brute force keep attempting to connect until it finally submits.
The final real sticking point before I could begin was more technical than critical. When I load in the image data it comes in 8 parts which need to be pasted together to form the final image. Unfortunately in Unity this was causing a noticeable pause each time this happened because it can’t perform any functions outside of the main thread. This almost forced me to move across to the Unreal Engine but instead I now just preload everything I need before I start riding. This has the advantage of guaranteeing the data will be ready when i need it but the downside is I can’t ride further than I originally wanted (not a big issue for me) and I need to wait about 10 minutes for a 30 minute cycle.
