Project "Konnektid"

Querying geo location based data with MongoDB

6 August 2012 | 22:20 | Galina Slavova

The most recent project I am having my hands full with is, I should say, a combination of challenging pleasure and work. And something completely new! Yes, this is my first time I've been asked to develop a web applicaiton in something different than Microsoft technology. This time I'm using open source technology to write a web client-server application. This time it is JavaScript on both sides, thanks to Node.js. And also thanks to Bart who has been a fan of this technology fo some time now.

This new application's name is Konnektid - a social skill sharing web application. It has its focus on finding for us, individuals, other people who we don't know, but who have the skill to help us ( or teach us ) in our personal or business accomplishments. This concept blends with the so called peer-to-peer economy, P2P.

A frame from the video produced by BrandFirends (their website coming soon).

Konnektid, v2

Bart delivered the first version of Konnektid and I am now responsible for the second one. One of the important features in the new version we just deployed last week ( Yes, we are live, and yes, it works fine on all modern browsers and doesn't break on IE 8! Champaign to that! ) is showing geo coordinates when a registered member searches for a candidate, based on a specific skill.

Here is a concised requirement to illustrate the feature: Whenever a registered member searches for a candidate based on specific skills, the produced result will show the closest candidates to this member's current location, or otherwise - unknown distance ( "? km" ). The geo search we restrict to be in a radius of 100 km.

Here is a visual of the search area inside Konnektid. A result will be produced either by clicking on a skill in the tag coud or entering a skill word or part of it:

For the qurious eye: go and register on Beta. Play around, crack it and call us to receive a chocolate bar!

The geospatial support which was needed to fulfill the stated requirements was not a part of the backend database of this first version of Konnektid - CouchDB. CouchDB is a NoSQL document oriented database with "precompiled" views. It reminds of the stored procedures in Microsoft SQL databse ( but then SQL server is a relational database management system - RDBMS ).

While examining different platforms with geospatial support that would match our case, I saw Greg Studer's video presentation on "Geospatial Indexing with MongoDB" and after discovering the easyness of installing and getting started with MongoDB and mixed geo-data queries, I was sold!

Here is a comparison table for open source databases with a geo spatial support.

data structures, indexing and queries in MongoDB

A short introduction to MongoDB:

  • A document-based NoSQL database managmenet system.
  • Documents are described in JSON or BSON ( binary serialization of JSON documents for better performance ).
  • Documents are, schema-free, but you could arrange structure and restrictions on them by using mongoose middleware.
  • Every property in the document object can be set with an index. The geospatial data must have the "2d" index.
  • It works with collections - a collection has documents of the same schema.
  • A database is actually a grouping of one or more collections.
  • A JavaScript shell interface comes with the installation of mongo, where queries can be run.
  • There is a native UI desktop application for the Mac to MongoDB - MongoHub. Very convenient to run it in development prime time.

Queries in MongoDB

To write queries in mongo is easy like breathing. ( Obviously they did their homework. ) To find a document according to a search filter, you type in the shell:

Don't mind the transparancy of the terminal's background. If you do, then it is MongoHub what you see behind.

This query says: find me in the candidates collection a JSON document whereby the value of its name property is equl to "Galina Slavova". The forEach(prinjson) is not a part of the query, I used it only to beautify the output, so I can put it in a frame for you to see.

Prior to running the query I typed: use myDatabaseName to make the database current and ready. The query filter itself is also in a JSON-like format: it is all key-value pairs inside curly braces {}. The result, as you see above, comes in the same format, too. One for all!

Now, I can enrich my filter by specifying more:

Example #1: > db.candidates.find( { name: { $in: [ "Galina Slavova", "Bart Riemens" ] } } );
Result: it will only find exact name matches if present in the array.

Example #2: > db.candidates.find( { name: { $regex: new RegExp("art", "i") } } );
Result: it will only find all documents with names containing "art".

Example #3: > db.runCommand( { geoNear : "candidates", near : [ 40, -73 ], num : 10 } );
Result: it will find all documents with locations near lon: 40 and lat: -73 and will reduce the result to 10.

There's a plethora of special keywords, which meaning you can guess: $lt, $lte, $gt, $gte, $in, $nin, $or, $not and many more.

There are many nice features of mongo and I think they are very well explained in the book: MongoDB: The Definitive Guide.

Geospatial searching with Mongoose and MongoDB

I used the mongoose library as a data access layer to MongoDB. Mongoose is a Node.js module, designed to work in an asynchronous environment.
There are different tutorials on how to use mongoose in one's Node.js project.

One important thing to know is that mongoose introduces a way to model your data through schemas. A visual below:

First define the document fields through a schema object. Then instantiate the model and give it a desired collection name.

After defining my document model to have a geospatial property lonLat, and populate the collection with data, I am ready to provide filter options and perform geo queries.
NB! In a normal geek jargon one would talk about latitude and longitude coordinates in this order. In mongo, you don't. So, remember to always put the "lon" value before the "lat" one in the array when doing a geo search! Also do not forget to implement an index on the geo property: "2d".

During the geo-search-with-mongoose-and-mongo POC for Konnektid I used the following code. The code is almost the same as the JavaScript queries in the mongo shell, which I showed above:

The values above are replaced by variables:

  • collectionName ( e.g. "candidates" )
  • locationCenter ( e.g. [ 4.6420051, 52.381530100001 ] )
  • maxResults ( e.g. 10 )
  • range ( e.g. in meters: 100 000 )
  • earthRadius ( a constant, in meters: 6378000 )

There are two parts of this function: providing the filter options as input, and then, writing the next behaviour in the callback function.

In the end, we integrated mongoose with mongoDB into Konnektid through the Crafity middleware framework, crafity-storage, in particular. Crafity-storage supports DAL to CouchDB and to MongoDB at this stage. Each one of them is supported by a provider. Here is the signature description of the method for geo search:

A repository inside Konnektid makes use of this provider method. Its method description:

All crafity modules are using open source libraries and are ready to be shared as such. They are all available here on GitHub.

One of my next blog posts will be dedicated to those modules, explaining them and providing with unit-tests.

Geo coordinates

And one last thing before we all leave the screens and go to have dinner!
When you decide to enter the realm of Konnektid, and want to see geo searching with dynamic distances, just do not forget to allow your browser's navigator to save you current location.

Do not use the private browsing window of Google Chrome for this purpose, or any of Opera or Internet Explorer versions.

Alright, this is pretty much the end of this story.
If you feel like commenting, asking or sharing with us, send us an email. We'll be happy!

Have a good one! :-)

  • Geospatial indexing
  • MongoDB
  • CouchDB
  • Mongoose
  • Node.js