ODK Collect

ODK Collect is definitely one of my favourite Survey Tools because not only is it open source and has a free to use mapping element (more later) but it is relatively simple to get started and can scale up as your skillset grows. This is the kind of open source project I love as it encourages people to engage with technology at their own pace and absorption rate rather than sticking to pen and paper or having specialist software created and when it is supported by the likes of Google with free personal cloud hosting, It just gets my creative learning juices flowing. In fact I love it so much I will add a simple get started guide but for the tech geek savvy, the full instructions can be found at https://docs.opendatakit.org/

As I mentioned before it is simple to get started and all you need is a google drive account. If you have a gmail account then you will have a google drive. If you have an android device then you should have set up a gmail account for the PLAY store at some point. If later you want to expand its capability then you can use an aggregate cloud service, which we will cover at a later date, google provide the service (personal) for free for a year, but for now lets just work from a windows 10 pc, google drive and android phone.

You can use any android device and I actualy performed a long survey and ran out of battery on my tablet so I carried on with my phone. The result was astounding and was as if it was all performed on the same device. I will add that I waited till I got home and uploaded to my google drive over the internet one device at a time rather than both together. I can imagine that simultaneous sync of multiple devices might cause it to get a little disjointed with an asset survey but if it was just a questionaire, sent to millions of people, it should be ok. I have also proposed that when performing this on a BIM asset Survey, each site should have a different survey which I will cover in more detail later.

LETS GET STARTED

Ok, So first things first, Power up everything and get connected to the internet and we will download the Windows Application. You can also download it for MAC here too.

https://github.com/opendatakit/build/releases/tag/0.3.5

Don’t worry if you don’t have a windows PC or Mac as you can also use the web hosted form designer here http://build.opendatakit.org/ but you will need to create a login and user account but it is exactly the same just web hosted.

You will need to scroll to the bottom of the page, where it looks like below and select which operating system you have (win.x64 is for 64 bit windows like windows 10 / 8 / 7 64bit / xp64) (win.x86 is for 32 bit windows like windows 7 32bit / xp 32bit / vista (why oh why are you running vista)) (macos is for macs).

You will now need to extract the zip file which is most likely in the downloads folder. I know ths is not best practice but I trust this file so I extracted it direct to my programs folder by the following process.

Right Click the ODKBuild.zip file, as shown below, and select extract all from the menu.

Now Navigate to Your Program Folder and click select folder button. For me it is as below

If all has gone well, you will now have a folder exactly the same as the zip file but without the zip on the folder symbol. If you open the folder, you should see an odkbuild application and icon. double click to launch the program.

Now the last bit is personal preference but I like to add to my start menu or Taskbar so to do this, Right click the application Icon and select Pin to Taskbar. Right Click again and select Pin to Start Menu as below.

You can now close your folder explorer and click on the icon on the taskbar to launch. Depending on the speed of your machine, it may take upto 30secs to launch.

CREATE YOUR FORM

Ok so we now have ODK Build installed and running, lets create our first form but what do we want to capture. Lets keep it simple with an asset description, an in service or decommissioned option, 3 Pictures including asset location / asset / asset dataplate, Geolocation Point and finaly space for comments.

It is worth noting at this point that there are many options here to explore later like geopaths / geo areas / video / audio, but this is an introduction at this stage and I urge you to self learn these other capture tools to streamline your surveying as they follow the storage and recall principles as the photos.

So Rule 1 – Save the blank form with a meaningfull name, this is important later, lets go with test1 for this exercise.

1. Click File
2. Click Save Form As
3. type “test1” in the file name box
4. Click Save (You will note that it saves as type ODKBUILD File)

Ok, So you now have you .odkbuild file, lets add some form data

So you will now have the below screen, pretty blank and boring, unimpressive

On the bottom bar, Click Text Once. It is highlighted below

You should now have the following very boring looking screen below.

Now In the Right hand column, start entering the following information as below in the picture below but a rough idea of which each box does is as follows:

• Data Name – This is the field name and is unique for programming functions, I would also suggest to keep it distinctive from database confusion so as this is a field entered free text value, I add _Field to the end of Description. This means that if the spreadsheet is processed programmaticaly, also called scripting, it is unlikely that it will be confused with another field in the database called Description. A handy tip for all spreadsheet work.
• Label – This is the field Name as displayed in the App to the end User. This is so you can accurately describe in human terms what you want.
• Hint – This is further information to describe what you want the end user to capture or do, any data quality requirements etc.
• Default Value – This is like AutoFill, It prepopulates an answer which can be handy as a time saver if 99% of answers are the same but I am cruel and want them to not get complacent. Data Quality and integrity is key for surveys
• Read Only – Prevents the field being edited. This might be to show a calculation result i.e. a risk index for a confined space entry form. Just discovered another use for this with health and safety.
• Required – Prevents the form being submitted or finalized until the field is completed.
• Required Text – This is the text to display to the user if they do not fill in this field.
• Length – This is where you can set the Min and Max character length of the field. Sounds pointless in basic forms but when you work with databases like SAP and ESRI ArcGIS, this is vital so you can script the results into them.
• For now don’t worry about the Invalid and advanced text features.

Ok, Pat yourself on the back as you just created your first field and got to grips with the fundamentals of survey forms. The other elements are derived from this same format and I will discuss the extra properties as they become relevant.

So next up is to create an option to flag the asset as in service or decommissioned / Out of Service But First learn the most important skill of saving your work by

File -> Save Form

Ok Now, Select the “Choose One” option from the bottom Add New bar

Now fill in as below, note that once you have entered option 1 click “Add Option” as highlighted to add a second option. Note the value boxes are the programmatic values and the English values are what are displayed to the end user.

File -> Save Form

Now Select “Media” from the Add New bar. We use the Media element/Field for Pictures/Video/Audio capture. Anyway back to the task in hand, Fill in the fields as below.

Note that the Drop down menu “Kind” is defaulted as “Image”, We can change this to Audio or Video from the Drop down List. Can you also see how The Boxes or Modules on the left are displaying the Label field / Hint field / Data Name.

Below I will create the next 2 Image capture modules. try and work out the field data from the module boxes. Note that the Asset Picture is a required picture so I have checked the Required Checkbox.

Now We need to know where the asset is. For this we have to consider what environment the asset is in, is it outside or inside or surrounded by metal structures. This is important when considering GPS only options. For more about how GPS works, and I use GPS as a generalised understanding of GNSS, please see my other posts.I always use the setting below as BIM environments can give errors in GPS locations.

Select “Location” from the Add New Bar and fill out as below. Ensure that Kind is set to Point and Style is set to Manual(No GPS). This gives the option of using GPS via the device location services when outside or manualy plotting on a Satelite map (Google Maps) or Open Street Maps if you know that the position can’t be trusted.

Now For our last Question for the Surveyor, Lets give the option to add any further comments.

Select “Text” from the Add New Bar and complete as below

File -> Save Form

Now You need to export the form as a spreadsheet and upload to your google drive.

So now do the Following

File -> Export to XLSForm

Now Log into your Google Drive and Upload the form there. We will need to open the file in google sheets by simply opening the file from your Google Drive and clicking “open with google sheets” as shown below

add a sheet for submissions by clicking the plus button next to the tabs as highlighted below

Right click Sheet1 tab and select move Left (this is houskeeping), Right click again and rename tab something like Responses

Now Highlight and copy the Responses sheet URL as below ( yours will be longer as I cut mine short for obvious reasons)

Now Go back to ODKbuild program and select as below

Settings -> Form Properties

now in Form Properties, paste the URL you copied from your Google Sheets Responses tab into the field marked Submission URL. Also fill in the field title on device to something you want the form to be recognised by. Give an Instance name as this helps in more advanced sessions as this can auto increment with a little code but for now something simple like “filed”. Then Click Done.

Once saved on your Computer, Upload to your Google Drive and this ends the setup portion of the form and ends this section with ODKBuild. If you have followed this writeup, you should now have 3 files in your Google Drive as below

You can now access the Survey XML by using the ODK Collect App downloaded from the android PLAY store (its free and has the clipboard symbol- beware there are many similar charged for apps).

Once you open the app, select the menu (3 vertical dots in the top right corner) and select General Settings.

Select Server and adjust the settings as follows (tap to open their selection menu)

Type – Google Drive,Google Sheets

Google account – you can select your account or Add account and enter your details of the google drive account you are using. you can have multiple accounts on one device so if it does not show the files later then check the correct account is selected.

Select the device back button to go back to General Settings

Select User Interface and adjust as follows

Mapping SDK – Google Maps SDK ( you can select open streetmap but it has no satellite)

Basemap – Satellite

Now Click the device back button

Select Form Management – Here you can adjust picture size and video resolution to save data and drive space, this is a test and we will leave as standard (Original and highest resolutions)

Select the device back button twice so you are back at the Main Menu

Click Get Blank Form Button

You should load into your google drive but if not, click My Drive in the bottom Right. Navigate your folders till you get to your Uploaded XML file. It will have a blue file icon. Click it (checkbox) and then click Download selected. You should get a dialogue box saying Success. Click OK.

You are returned to the Main Menu so now select “Fill Blank Form” Select your form and complete the form. You can progress through the form by swiping left or by skipping using the arrow drop down menu.

When you get to the geopoint, note that if you have a GPS signal, it will locate and give you an accuracy based on the devices location services. If you are not happy with this you can long press on the screen at the desired location on the map to assign a new position. Note that manual positions have an Altitude of zero and no accuracy data for obvious reasons. Once you are happy with the position, select the disk icon to save it

Once you reach the end of the form, you can name it, Note that it is defaulted to the Name on the Device. I always change this to the description of the asset so if i need to check it later then it is easy to find when there are 150+ forms on the device.

Make sure the form is marked as finalized

Click Save Form and Exit – You should get confirmation of Succesful save.

You can send thes in immediately and there is an option in the app to automaticaly send them in once finalized but I prefer to save Data and Send the forms over WiFi. 150 forms with 3 full size phots each is about 1.1GB so its something to consider.

When you are ready to send the forms, Click Send Finalized Form, Click select All (you can select individual if something was urgent) and then Click Send Selected.

These will all now be uploaded to the spreadsheet in Google Drive. The Photos and Media files will also be uploaded there too with hyperlinks.

On the app you should have Upload Results – Success.

Click OK

Congratulations you have now performed a survey.

LETS VIEW ON GOOGLE DRIVE

OK, if you now open your google sheets file with the responses tab, you should see your data as below with hyperlinks to the photos, Lat Long in decimal degrees, Height and accuracy of the GPS position and the comments etc.

This is great to have the data in one place but if you want to navigate the photos as if on your computer you have to be tricky as they are not in a file as with most cloud based files now.

Google Drive has a greate search function though that allows you to do this.

Remember the name of the XML file you created, we called “testXML”, well this is used as part of the file name of the uploaded files related to that survey.

Simply type this filename into the search box at the top of your Google Drive and it will display all those results but if you select the drop down arrow at the right of the search box, you will get an in depth search function. In the image below I have also clisked on the dropdown arrow next to “Type” as I want to select Photos & images as the file type. This will help navigate in line with the returned survey spreadsheet data for someone to update the company asset list for instance. It is also possible that you can upload Audio and Video files which will help give a deskbased data entry clerk perspective of the site and these can be search filtered to prepare for the deskbased exercise in the same way.

Ok now in the section “Includes the words” enter the XML file name, in this case “testxml”, it does not need to be case sensitive. not how the search box is auto populated with the correct search criteria. Now Click Search. As you get more familiar with this technique, you may find it marginaly faster to type the code directly into the search box and press enter for search but I like GUI methods. See Image Below.

So the Enlarged image of the first picture is as below, and this corresponds with the first image on the first Spreadsheet Line on the Responses Tab that you created. I find that navigating using the right arrow key on my keyboard works easier than clicking the arrow on the screen to move forward through the images but upto you.

A couple of things to mention, first the filename is displayed in the top left corner. I have edited mine for obvious reasons but it shows it has collected from the device, the XML file name, Date and time of capture ( yes this was after midnight and as Arnold Schwarzenegger sais – Sleep Faster) and a file name of the original jpg.

Second, You can also open this file for editing or connecting into Connected apps by clicking Open With in the top centre of the screen. These apps can be created by your own developers or use many of the free or commercial apps listed for sharing geotagged photos. One such app is GeoEditor by maptiler. This is free for domestic albeit with a watermark but there are reasonable paid plans and desktop or cloud services but the point is you can link or upload the files from your google drive using the gps matadata. A quick little warning about this metadata is it will be as recorded at the time of photo so not only will it be maybe 5ft from the object if taken at distance, but also if taken inside the building, it will be a completely inaccurate result as GPS only works outside or sometimes inside GRP structures but concrete and metal structures will not give line of site to satellites.

A better approach will be to csv the spreadsheet and load it into you mapping service and leave your photos on the google drive as the link is a web link not a local one. Obviously you want to back up your photos so I would recommend storing them by downloading the zip file and storing on a usb drive which are cheap, small and can be kept in a fireproof safe.

Obviously the above would require a commercial google account for companies and enterprises and the usual legal use bits and bobs but it does get around the need for a cloud computing service and keeps everything on a tight budget while complying with GDPR, as you are collecting data and not exposing held or sensitive data via external link to the internet which should also satisfy security measures as google Drive commercial is secure which is another complication you don’t have to worry about.