Setting up Nodes

Setting up Nodes

I’ll start off by demonstrating how to set up a node. The node I use is a Raspberry Pi, but the setup would be the same on any other Linux microprocessor.

The folder set up is as follows. When the video/image is being recorded, it is being written to a ‘sync’ folder. Once the file is closed, it is moved to the ‘buffer’, from where it is copied onto a ‘buffer’ folder on the external storage (SD card). Once this is done, the files are moved to the ‘uploaded’ folder.

First let’s create these folders.

cd
sudo mkdir flashair
cd flashair
sudo mkdir sync
sudo mkdir buffer
sudo mkdir uploaded
cd
sudo chmod -R 777 /home/pi/flashair

The last line gives everyone full control over the folder and its subdirectories. While this isn’t ideal in terms of security, it helps us in the sense that we aren’t worried about not having the needed rights to write to the folder.

1
Figure 1: Flashair folder and subdirectories

To carry out the motion detection and video/image recording we make use of motion. If you aren’t sure on how to set up motion, have a look at our previous blog article, http://sjj.azurewebsites.net/?p=701.

We will now edit motion’s configuration file to get the desired data. The changes made from the original .config file are:

daemon on #default off (This allows the motion to run in the background)
framerate 30 #default 2 (increased framerate)
width 640 #default 320 (changed width to match that of the webcam)
height 480 #default 240 (same as above but for height)
threshold 2000 #default 1500 (an event is considered as motion if 2000 pixels have changed)
pre_capture 2 #default 0 (captures 2 frames before motion was detected and adds that to the videos to make them smoother)
post_capture 5 #default 0 (same as above but captures frames after)
output_normal first #default on (this only stores the first image when motion is triggered)
ffmpeg_video_codec msmpeg4 #default swf (msmpeg4 is accepted by windows media player, hence easier to play on Windows) snapshot_interval 300 #default 0 (a snapshot is taken every 5 minutes)
target_dir /home/pi/flashair/sync #default /tmp/motion (changed the directory where videos will be stored)
snapshot_filename %Y%m%d%H%M%S-%v-snapshot #(change the format of the name of the snapshot files)jpeg_filename %Y%m%d%H%M%S-%v-%q #(change the format of the image files)movie_filename %Y%m%d%H%M%S-%v #(change the format of the video files)webcam_maxrate 5 #default 1 (increase the max framerate on lie stream)
webcam_localhost off #default on (allows you to set up a live stream of the webcam)

We also want to create a message (notification) once a motion event is detected and another when it ends. We also want to move the completed videos/images to the ‘buffer’ folder once they have been written. We need to write some python/bash scripts in order to achieve this, and then run them using some of the properties that motion allows us to use.

Scripting

The first script to be written is a python script to create a message when motion is detected. Install python on your device if you haven’t done so already. Now would also be a good time to go over the blog article on using the Python Azure SDK <INSERT LINK>.

Create a folder called ‘scripts’ in the home directory.

cd
sudo mkdir scripts

The python script used is as below:

#!/usr/bin/python

# Steven Johnston johnston.ac
# Umang Rajdev unr1g12@soton.ac.uk
# 29/08/2014

# Python script to create messages when a motion event is triggered.

# Import required libraries. Ensure Azure Python APK is installed
from azure.servicebus import *
import sys
import cPickle
from uuid import getnode as get_mac
import time

# Set all the variables needed for the message
args = sys.argv
# msg is the variable given to the script. This will be the message body.
msg = Message(args[1])
# Mac address used to distinguish different nodes
mac = get_mac()
# Serial used to distinguish between different sensors/webcams on the same device
# Need to find a better way to get a unique identifier for sensors connected
serial = '1'
# clock used to obtain the time the message was created
clock = time.time()
y =  time.localtime(clock)
print y
localdate = str(y.tm_year)+str(y.tm_mon)+str(y.tm_mday)
# custom properties set so that approprite subscriptions can be created
msg.custom_properties = {'mac_address' : mac, 'serial' : serial, 'time' : clock, 'localdate' : localdate, 'time_str' : repr(clock)}
print msg.custom_properties
# convert the message using cPickle so that it can be written to a file without losing any properties
x = cPickle.dumps(msg)
# write to a file
with open("/home/pi/flashair/buffer/message_log.log", 'a') as w:
  w.write(x)

Save this in the new ‘scripts’ folder that we just created.

Another thing we want to accomplish is to have messages every time a video/image file is created. Hence, we create a script that writes a log of the files that have been created by the node and need to be uploaded at the hub.

This is the python file, log_blobs.py:

#!/usr/bin/python

# Steven Johnston johnston.ac
# Umang Rajdev unr1g12@soton.ac.uk
# 29/08/2014

# Python script to create messages when a motion event is triggered.

# Import required libraries. Ensure Azure Python APK is installed
from azure.servicebus import *
import sys
import cPickle
from uuid import getnode as get_mac
import time

# Set all the variables needed for the message
args = sys.argv
# Mac address used to distinguish different nodes
mac = get_mac()
# Serial used to distinguish between different sensors/webcams on the same device
# Need to find a better way to get a unique identifier for sensors connected
serial = '1'
# clock used to obtain the time the message was created
clock = time.time()
y =  time.localtime(clock)
print y
localdate = str(y.tm_year)+str(y.tm_mon)+str(y.tm_mday)
# the file_details below are the custom properties that will be set once the message is sent
file_details = (args[1], mac, serial, clock, localdate, repr(clock))
# convert the message using cPickle so that it can be written to a file without losing any properties
x = cPickle.dumps(file_details)
# write to a file
with open("/home/pi/flashair/buffer/blobs_log.log", 'a') as w:
  w.write(x)

You will realise that this is similar to the previous python file.

Just to clarify, this is the procedure that follows on each node. Motion is detected, which triggers the software to start recording a video. At this point, a message to be sent to the service bus is also created, and written into the message_log.log file. Once the motion event ends, the video file is completed, and moved from the ‘sync’ to ‘buffer’ folder. Another message saying that motion has ended is also written to the message_log.log file. The details of the recorded video file are also noted in the blobs_log.log file, which will be used to upload it to the service bus.

Let us set up the script that moves the file from the ‘sync’ folder to ‘buffer’ once the video/image is written.

cd
cd scripts
sudo nano motion_upload.sh

This will open a new file called motion_upload.sh. This is what I have in my motion_upload.sh file:

#!/bin/bash
echo "working"
File="$1"
echo "File is $File"
mv $File /home/pi/flashair/buffer/
python /home/pi/scripts/log_blobs.py $File
echo "Done"

The first line tells the terminal to use bash to run this script. The ‘echo’ lines are only for debugging purposes. $1 is actually the input that the file receives. As you will see later, we will ask motion to provide this script with the name of the video/image. Hence ‘File’ is actually the file path of the newly written video/image. We then move this file from its current file path (which is in the ‘sync’ folder) to the ‘buffer’. We then invoke python and run the log_blobs.py file which writes the details of the file to a log file, which is later used by the hub to upload the file to the service bus.

Save and close the file. We now want to make it executable, so run the following command:

sudo chmod +wrx motion_upload.sh

Completing motion setup

Now that the scripts are ready, we should alter the motion trigger commands in the motion.conf file:

sudo nano /etc/motion/motion.conf

Scroll down to the External Commands section, and alter the following:

# Command to be executed when an event starts. (default: none)
# An event starts at first motion detected after a period of no motion defined by gap
on_event_start python /home/pi/log_messages.py motion_detected_at_%Y%m%d%H%M%S-%v

# Command to be executed when an event ends after a period of no motion
# (default: none). The period of no motion is defined by option gap.
on_event_end python /home/pi/log_messages.py motion_ended_at_%Y%m%d%H%M%S-%v

# Command to be executed when a picture (.ppm|.jpg) is saved (default: none)
# To give the filename as an argument to a command append it with %f
on_picture_save /home/pi/scripts/motion_upload.sh %f

# Command to be executed when a movie file (.mpg|.avi) is closed. (default: none)
# To give the filename as an argument to a command append it with %f
on_movie_end /home/pi/scripts/motion_upload.sh %f

Transporting data from node to hub

At this stage, we have now completed setting up motion. We now want to set up the 2 transport options for the nodes. This refers to setting up use of the flashair wi-fi enabled card, and the automation of copying files from the nodes to the hub through an external storage drive.

Option 1: Copy via USB

I’ll begin by showing how you can automatically have the node copy the files from the buffer to a folder on your external storage, every time you plug in a flash drive or a memory card. In my case I have it working for a flash drive, but there’s nothing to stop you from using any other medium.

Plug in your flash drive, and type dmesg in the terminal. You should see a few lines indicating that a new device has bend detected. You need to look for the line that specifies what the address of the device is. In my case it was ‘sda1’. It is usually ‘sda’ followed by an integer depending on how many devices you have plugged into your node.

I would also recommend giving your USB drive a name. I have mine labelled as “RSIS”. This just helps to identify the device and ensure that files are only copied when a drive with that label is inserted. To check what your USB drive is called, run:

sudo blkid

Take note of the LABEL of your flash disk.

Udev in Linux allows users to specify rules that are run whenever an external drive is plugged into the system. What we shall do is run a script that copies files from the buffer folder to the flash drive.

cd
cd /etc/udev/rules.d
sudo nano 99-input.rules

Now add the following line to the bottom of that file:

KERNEL=="sda1",ACTION=="add",ENV{ID_FS_LABEL}=="RSIS",RUN="/home/pi/scripts/copytousb.sh"

Swap ‘sda1’ for the address of your device. Also change ‘RSIS’ to the label of your device.

Note that in some systems, you may have an alternative file to 99-input.rules, in which case you will have to try by editing that file instead and checking if it works.

Finally, we write the scriptcopytousb.sh that we referenced above.

cd
cd scripts
sudo nano copytousb.sh

Below are the contents of copytousb.sh:

#!/bin/bash
touch /home/pi/test
sudo mount -t vfat -o uid=pi,gid=pi /dev/sda1 /media/RSIS/
sleep 10
mv -v /home/pi/flashair/buffer/* /media/RSIS/buffer/
touch /home/pi/test2
sleep 10
sudo umount /media/RSIS

The ‘touch’ and ‘echo’ lines are again just for debugging reasons.

The script first mounts the flashdrive to a folder /media/RSIS/. After a 10 second interval (to ensure it is actually fully mounted), we move the files from the buffer to the flashdrive, after which the drive is unmounted. Save the file and make it executable as we did previously (using chmod +wrx).

We need to also create the /media/RSIS/ folder.

sudo mkdir /media/RSIS

Option 2: Wifi SD Card

This is the more interesting and exciting implementation of remote data collection. In this case, we have a UAV come flying in to each node and wirelessly collect the data using the Toshiba FlashAir card. We plan to implement a situation where the card can be powered using an induction charging kit at the nodes.

For now, let us set up the scripts needed to upload files from the buffer to the wifi card. First, make sure you have the card set up i.e. have configured your wifi to connect to the card automatically. Next, we need to make sure uploading is enabled in the card. To do this, plug in the SD card in another computer and open the CONFIG file located in the SD_WLAN folder.

Change the contents to:

[Vendor]

CIPATH=/DCIM/100__TSB/FA000001.JPG
APPMODE=4
APPNETWORKKEY=********
VERSION=F19BAW3AW2.00.03
CID=02544d535733324708e664c47800e101
PRODUCT=FlashAir
VENDOR=TOSHIBA
UPLOAD=1
MASTERCODE=000f55bbcc96
LOCK=1
APPINFO=0000000000000000
APPAUTOTIME=1800000

The main thing to ensure is that UPLOAD=1 has been set.

Since someone has already written a brilliant python script to upload files to the wifi card, we are simply going to use that instead of having to write our own, which would have otherwise been a project of its own.

Go to http://www.extrud3d.com/flashair and navigate to the bottom of the page where usage of the script is explained. I would also recommend reading the whole article as it explains how the whole process works and provides more details on the operation of the card.

Download the zip file and extract it. Next place the flashair_uploader.py script in the scripts folder in your home directory. You will also have to install the ‘Poster’ library for python, as is mentioned in the article. This can be done by downloading the .tar.gz file from https://pypi.python.org/pypi/poster/ and extracting it. Navigate to the extracted folder, and run “python setup.py install”. For more on installing python libraries see https://docs.python.org/2/install/.

We now have the uploader working. The next step is to initialise how we will invoke the uploader to upload files from the buffer to the card, every time the node connects to the FlashAir card.

in Linux, there is a directory where all scripts are run whenever a network connection goes up. Navigate to:

cd
cd  /etc/network/if-up.d

Any scripts in this folder will be run whenever a network is connected. We will create a script called ‘uploadworking’ and place it in here.

sudo nano uploadworking

The script is as below:

#!/bin/bash
Buffer=/home/pi/flashair/buffer/
Uploaded=/home/pi/flashair/uploaded/
Python_Uploader=/home/pi/scripts/flashair_uploader.py
sleep 5
iwconfig wlan0 | grep "flashair" > /home/pi/debug.txt
if iwconfig wlan0 | grep "flashair" ; then
        for file in $Buffer* ; do
        echo "$file"
        sudo python $Python_Uploader -v -c http://flashair "$file"
        sudo mv "$file" "$Uploaded"
        touch /home/pi/testupload
        done
else
        touch /home/pi/testploadfailed
fi

Once again, the ‘echo’ and ‘touch’ files are for debugging.

We first specify the paths of the files we want to copy, as well as the path of the python uploader script.

Next, we have a 5 second sleep timeout to ensure that the wifi is connected and is not in the middle of configuration/setup. After this, the if statement checks whether the network we are connected to ‘flashair’ which is part of the SSID of the wifi network the card sets up. If we are, then the script runs the python uploader, and moves the files from the buffer to the uploaded folder, which essentially means the process is completed. On the other hand if we aren’t connected to flashair, no action is taken.

Make sure you save and close the file and make it executable using chmod.

3
Figure 3: the uploadworking script in the if-up.d folder

This completes the setup of nodes. It seems quite complex, but bear in mind that all we are doing is trying to automate the process as much as possible. If you follow each of the steps above in order, you should be good to go. Also, once the nodes are set up, the hub is not much more to add on to that.

Leave a Reply

Your email address will not be published. Required fields are marked *