Indoor positioning has failed to become ‘mass-market’. And we think one of the reasons is the difficulty in assessing the suitability of different solutions for any particular use case.
An obvious performance measure for indoor positioning solutions is accuracy. But it’s not the only important consideration. And fixating on a single number - the accuracy in metres - is likely to be very misleading. Some of the reasons why include:
- Accuracy can vary greatly from one environment to another. It’s important to know whether the quoted accuracy was achieved in strictly controlled ’lab’ tests, or in a busy, complex, real-world environment.
- Even in the same environment changes in conditions such as how crowded they are, will affect accuracy. An indoor positioning solution could be calibrated and tested under identical conditions. And then perform signficantly worse once conditions change.
- Accuracy varies across different mobile operating systems, different phone models, and even different phones of the same model. Calibrating and configuring an indoor positioning solution using the same phone will give a very different accuracy figure to averaging across tests with different phones
- A single figure in metres can mean different things. Try asking whether it’s one or two sigma, or RMSE, and for a single test, or averaged over a number of tests. And how the average was calculated.
Alongside accuracy, there are other important performance measures that need to be considered in selecting an indoor positioning solution:
- Frequency of updates can be just as important as accuracy in metres. If you know precisely where someone is, but only every 30 minutes, that may not be as useful as knowing where they are to 10m accuracy every second.
- For indoor navigation, positioning is only needed while apps are in use, in the foreground. But for most other uses from marketing to analytics, data must be gathered while apps are the background. Surprisingly many products that are advertised as marketing and analytics solutions don’t perform well in the background.
- Cross handset performance is critical. Indoor positioning solutions that only work on iOS, or only perform well on handsets from a particular manufacturer aren’t usually viable.
- Infrastructure requirement is also critical. Any performance measures must always be considered alongside requirements for extensive and expensive infrastructure.
We want to be completely open about our indoor positioning accuracy, and we’re going way beyond publishing a single accuracy number. If you want a single number - our typical accuracy is around 4m. But we don’t believe that tells you very much.
The image above shows a test we did at a real deployment of our technology, in a busy and complex exhibition environment. The joined dots show the positions calculated by our software. We think visualisations like this give a far better indication of performance, and suitability for a particular use-case, than a single accuracy number. Surprisingly we haven’t come across a single other provider who has published a visualisation like this.
- The test is in a real-world environment. It’s not a lab test designed to give the most favourable results
- Crowd Connected’s indoor positioning solution doesn’t require any fingerprinting or calibration. So the results are typical of what might be achieved for a real user’s phone.
- Crowd Connected’s indoor positioning solution adapts to environmental conditions, so similar results will be achieved in an empty or a crowded hall.
- The visualisation shows how well the data can detect visits to small areas like exhibition stands - and therefore how valuable it is for marketing or analytics.
- The dots show the frequency of updates as well as the accuracy of the indoor positioning.
We’ll be publishing the results of many more tests in future, and will always try to be 100% open about the performance of our indoor positioning technology.
We encourage every other provider to do the same. And if you come across a provider who simply says their system is ‘2m accurate’ and is otherwise completely silent about performance, start asking some questions.
