Automated Drone Missions on iOS

Exploring DJI Drone SDK

First Contact

Having DJI Drones at Tooploox we’ve finally come up with the idea of playing with the SDK instead of only flying. It’s strange the idea appeared only recently but better late than never. I find it amazing how drones (mainly DJI) stormed the mainstream consumer market. What seemed like a futuristic tech just a few years ago has become a common tool. There are also many possibilities in the professional market e.g. in mining, agriculture, insurance or construction.

We kicked off Tooploox Labs project to be ready for anything our clients might want to do with drones. For me, it was also the first time I was flying one. Yes, I had never controlled one before despite having the access.

General

If you never piloted a drone, don’t worry – it’s very easy. Just make sure you have a controller and a nice weather. With a controller, you can be sure your drone will not fly away in the direction of the setting sun. Technically you can steer it from your phone but it’s not recommended. The phone signal is weak and if it ever gets lost, you might lose your drone. The modern drones handle themselves pretty well even in not so perfect weather conditions but on a windless day, you will save yourself a lot of nerves. Last but not least, make sure to turn on collision detection and fly straight in the tricky places as the sensors work best that way. I don’t know about other brands but DJI drones are quite advanced and very hard to break if you follow these steps. The piloting is very easy and the drone couldn’t be more responsive.

So, enough about flying, let’s go into more technical stuff. But before we jump into the code, we have to carefully check what features from SDK you want to use with the drone model you have (or plan to acquire). We described our feature set before we checked the documentation deep enough. We couldn’t make our Spark accept the waypoint mission. On the product page, there are only features which Spark has, but it’s very hard to check which ones it doesn’t. The SDK errors were very brief and opaque but finally, we realized that Spark doesn’t have the waypoint features. We ended up buying Mavic Pro.

Another not-so-obvious thing is simulating the drone. Remember, you can’t simulate the drone you don’t own. The simulator which is part of the DJI Assistant app on macOS requires that the drone is connected to your computer. The software itself makes the drone think it’s actually flying (without spinning rotors of course) but every other sound and signal looks like it’s really operational. In the simulator, you can pilot the drone through an empty 3d area and mock coordinates of your choosing.

Dronoway

The initial idea for Dronoway, the outcome of our experiment, was to conduct automated drone missions. First of all, we chose an iOS app to control our Mavic and machine learning part (embedded in the app) to decide if the machine has enough battery to conduct the mission. If not, it should return home and continue the plan from where it stopped. Finally, we decided to break it into 2 parts – the app and machine learning as it was too much for a two-week plan. Konrad, our designer, jumped on board to make sure UI and UX are crisp and functional.

At last, we decided on the following features:

– Connecting to Mavic

– Showing video feed from the drone

– Showing the drone position and heading on a map

– Setting up the waypoint mission

– Editing the waypoint mission

– Running the waypoint mission

– Stopping the waypoint mission

SDK

If you are interested in what you can develop with the DJI drones, here is the link to the documentation. If you feel my descriptions and code samples are lackluster, you can find more info there.

The SDK itself consists of:

– DJI iOS SDK – SDK with under the hood functions for controlling a drone. Available as Cocoapod.

– DJI iOS UX SKD – a collection of UI components as seen in DJI GO 4 app. There are free to use inside your application. Available as Cocoapod.

– Video Previewer – exemplary application with a drone video preview feature. It happens to contain the code needed for the feature unavailable in the base SDK. It means you have to download it from GitHub and add to the project yourself.

Initial Setup

Before you even start coding, there are a few formalities you have to do. You have to register or login to your DJI developer account and create an app there. The bundle id you need to proceed is very important and if you ever change it inside the app, it will stop working (happened to me, wasted some time on it). With a project created on the DJI website, you can jump to the Xcode.

Connecting

Connecting to the drone is easy. You just have to call DJISDKManager.registerApp and pass the object which conforms to DJISDKManagerDelegate as a parameter.

Here is a simple implementation of the DJISDKManagerDelegate.

Waypoint

I won’t cover the whole logic of adding points and handling the Map View. There are plenty of tutorials on that topic. But if you have the desired coordinates and your drone is capable of the Waypoint missions, you can perform it now.

This method contains everything you need to start and conduct a Waypoint mission. Of course, it’s just to get started and you can do much more. I will go through important things here, some of which took me a while to figure out.

A mission has to be with minimum 2 points. A single point cannot be a mission.
Clear the remains of other mission and create a new one.
Enumerate custom `mission` structure data and extract the needed information, mainly about the coordinates. Then create waypoints for the mission. Here we can set some specific waypoint parameters like altitude but also much more like camera behaviour etc.
Setup mission parameters. Here we can set some general options like aircraft heading mode, speed and finish action.
Check if mission parameters are ok and the mission is possible.
Load mission to memory and prepare for the upload.
Add a listener on upload event. A lot of SDK is based on listeners and functions returning errors. The API is not Swifty at all and needs some figuring out to make it work. The listener is needed to debug if uploading fails (like when we try to upload waypoint mission on incompatible drone) and to know if everything is ok. When the mission is uploaded, we can start our drone.
Here we actually upload the mission to a drone. Or at least we try.

As you could see, it’s not very complex but there is a lot of things we have to do and the API could be a lot better. But to be honest, it gives us the ability to control the drone from our iOS device. And it couldn’t be better than that.

Video Preview

A video preview view was a tricky one. It looks like it should be just part of DJI UX SDK as it’s just their view. Not at all. You have to download and add `VideoPreviewer` from DJI to your project. You can find the full guide here or stay with me for a brief (and Swift) description. Note: my goal was to see what the aircraft can see on the iPad screen so I didn’t implement all the camera functionalities like shooting photos or videos. When VideoPreviewer is correctly imported into your project and you chose a view which displays it, you need just a few lines to have the preview ready.

And then your class (View Controller for me) have to conform to

DJIVideoFeedListener protocol. Here we just make sure we update our view with a most recent set of display data.

Testing

The testing of our Dronoway app is another story. We could test either with simulator from DJI Assistant app or in the real world. As I mentioned, in both cases you need the actual drone and you cannot use the iOS simulator. Also, Mavic Pro controller will take your iPad lightning port so make sure you have wireless builds enabled or it will be very hard to debug your app with Xcode.

Testing with simulator works fine. You can easily simulate any location. Just make sure you simulate the drone location close to you and your iPad in the real world. I spent some time trying to debug something as simple as ‘your drone is too far away’ (and API will say it’s just some general error). But overall, it’s hard to do serious testing. Oh, and people working in other rooms will come to check out if your laptop will blow up soon. It’s not your laptop, it’s the drone cooling system – it will turn on eventually. Very annoying if you work in an open space.

Testing in the real world gives you a better data but it’s pain to debug this way. I was literally sitting with my MacBook in the high grass rebuilding over and over. But I was only able to do it because I was in the field where most bugs came up. I must say, it’s pretty satisfying to test an app this way. You build, connect to the drone and it files. Time-consuming – yes, but I had a blast! Remember, the Mavic Pro can fly as fast as 65 km/h so be careful with your waypoint. I waited for a nice weather and made sure my waypoint altitude was higher than the obstacles (hence 40 meters in my example), at least for debugging purposes.

Conclusion

So, our Tooploox Labs project succeeded. We did our fair share of research and development and familiarised a lot with the DJI SDK. Personally, as an engineer who loves to work under the hood, I had a lot of fun with flying and programming drones to do my bidding. If you have a DJI aircraft and some free time, you should totally check it out. Below you can find the video of the working app.https://www.youtube.com/embed/eVH0pQPEdJo?feature=oembed&enablejsapi=1&origin=https://www.tooploox.com,