A technical explanation of the SARTrack Client-Server
system, including the issues of on-going Updates and SARTrack Versions.
For
a long time SARTrack consisted of a Windows based program, which was
mostly designed to do ‘live tracking’ of SAR Teams in the field, using
the Amateur Radio APRS tracking system, and later also other systems
like GPS enabled radios.
To be able to share this data, SARTrack used the APRS-IS Internet network to share this data between SARTrack computers, and this includes various non-standard features like the Operations Log and more.
Using the APRS network extremely limited the possibilities.
So I started looking at using existing Database software like SQL based systems. And they had to be free, because SARTrack is free, and I get no income from it.
After wasting 9 months trying, I discovered it simply could not be done.
The problem is that SARTrack works with live data, which is then pushed to all connected SARTrack clients immediately when it comes in.
SQL based systems work on a request basis. The connected clients get NO data at all, until they ask for an update. And the request time involved (in milliseconds) is so big it cannot cope with any type of live data.
So I ended up writing a complete Database system from the ground up. And it is unique in the fact that:
- It pushes (almost) all incoming data to all connected (and logged in) Clients;
- It uses highly compact database structures, to be able to send its data over very bad Internet connections (Satellite, Mobile phone), while SQL databases convert all data into text strings which are huge.
- It is capable of working without Internet access, due to the very unique way of being able to use Local Database Servers, which themselves are also Clients which can connect to an Internet based Master Server as you can see here.
So how does it work?
When a SARTrack Client or SARTrack Local Database Server connects to a (Master) Server, the Server will check the Version Number of the connecting Client. This must be in the currently allowed range, e.g. not too old. See below the reason for this (and the problems involved).
The Client will then try to log in. When is succeeds, the Server will send its Date &Time stamp to the Client. From that moment on, all SARTrack Date & Time is based on the Server time, no longer on the local computer time (so the Date and Times you see in the SARTrack client, are not based on the local PC time). If the time difference between the local PC and the Server is too big, an error will occur.
The Server will send to the Client the current Active Operation name.
The Client will then request a full update of the complete database records for this Operation. The Server will compile a single data Stream which includes all database tables and transmits this as a whole to the client. This contains all Stations, Objects, Messages, Log entries, Teams, Tasks, People, Equipment, Journals, etc., etc.
Once it has arrived, the SARTrack Clients can display all this, and the Server will transmit any updates to the Client live when they come in.
In case of a Local Database Server which has received this stream from the Master Server, it will compare all data with what it already has stored itself and update all records with a later timestamp. It will then send back to the Master Server all records which this Server did not have or which where to old.
This is the Server Synchronisation system, which keeps the data between all Servers identical.
All this requires that the database structures between Servers and Clients are identical.
Up to a certain point, the Server can deal with a Client which is of an older version, because all database tables have a ‘minimum version’ number attached, and in addition also an internal version number.
This means that on initial connection, the Server will skip (not add to the Stream) any database tables which it knows the Client cannot deal with. So far so good.
All further data transmissions between Clients and Servers are packed into a small Stream with a standard header, followed by whatever the Client is transmitting.
The Server will copy this Stream to all connected Clients (and in case of a Master Server, to all other connected Local Database Servers).
Any Client or Server which is too old, and does not recognise the Stream, will drop it with an internal error.
Basically, you do not want all this to happen, all Clients and Servers should always be up-to-date.
Unfortunately, because the SARTrack system is still in heavy development, new features (requested by you) are being added all the time, and therefore also new Database tables.
The real problem occurs when a structural change is required with an existing Database table. I try very hard to avoid a situation like that, but sometimes I simply cannot get around it, when new features require that internal changes must be made in existing tables, to make the new feature possible at all.
When this happens, any Server could still deal with a connecting older Client, because it knows its Version number, and could still transmit the older version of the table.
Sadly, this principle does not work when multiple Database Servers are linked together, as is the recommended way of using SARTrack.
Because, when an up-to-date Client transmits a new table format into a Stream to the Server it is connected to, this Server re-transmit this Stream to any other connected Servers, which in turn distribute it to their own range of connected Clients, and they have no control over the contents of the original Stream.
At this point, any Clients (including Local Database Servers) with a too-old Version will fail with a ‘corrupted Stream’ error.
And all this means that in a situation like this, all Database Servers and all SARTrack Clients must be updated to the latest version at the same time.
And this includes the Master Internet Servers from SARTrack Limited itself, the Germany based one and the New Zealand based one!
So I do my very best to avoid this situation.
Recommended practice:
When you start a new Operation or training, always download the latest Server, Client and Forms packages from the SARTrack website, and copy them onto a USB stick. Takes this with you to location, and update any Servers and Clients as required.
Any Internet based Master servers can be updated by clicking on the Server Monitor icon in the Windows Icon Tray, and selecting Check for Updates.
25 November 2017
Bart Kindt
CEO and developer
To be able to share this data, SARTrack used the APRS-IS Internet network to share this data between SARTrack computers, and this includes various non-standard features like the Operations Log and more.
Using the APRS network extremely limited the possibilities.
So I started looking at using existing Database software like SQL based systems. And they had to be free, because SARTrack is free, and I get no income from it.
After wasting 9 months trying, I discovered it simply could not be done.
The problem is that SARTrack works with live data, which is then pushed to all connected SARTrack clients immediately when it comes in.
SQL based systems work on a request basis. The connected clients get NO data at all, until they ask for an update. And the request time involved (in milliseconds) is so big it cannot cope with any type of live data.
So I ended up writing a complete Database system from the ground up. And it is unique in the fact that:
- It pushes (almost) all incoming data to all connected (and logged in) Clients;
- It uses highly compact database structures, to be able to send its data over very bad Internet connections (Satellite, Mobile phone), while SQL databases convert all data into text strings which are huge.
- It is capable of working without Internet access, due to the very unique way of being able to use Local Database Servers, which themselves are also Clients which can connect to an Internet based Master Server as you can see here.
So how does it work?
When a SARTrack Client or SARTrack Local Database Server connects to a (Master) Server, the Server will check the Version Number of the connecting Client. This must be in the currently allowed range, e.g. not too old. See below the reason for this (and the problems involved).
The Client will then try to log in. When is succeeds, the Server will send its Date &Time stamp to the Client. From that moment on, all SARTrack Date & Time is based on the Server time, no longer on the local computer time (so the Date and Times you see in the SARTrack client, are not based on the local PC time). If the time difference between the local PC and the Server is too big, an error will occur.
The Server will send to the Client the current Active Operation name.
The Client will then request a full update of the complete database records for this Operation. The Server will compile a single data Stream which includes all database tables and transmits this as a whole to the client. This contains all Stations, Objects, Messages, Log entries, Teams, Tasks, People, Equipment, Journals, etc., etc.
Once it has arrived, the SARTrack Clients can display all this, and the Server will transmit any updates to the Client live when they come in.
In case of a Local Database Server which has received this stream from the Master Server, it will compare all data with what it already has stored itself and update all records with a later timestamp. It will then send back to the Master Server all records which this Server did not have or which where to old.
This is the Server Synchronisation system, which keeps the data between all Servers identical.
All this requires that the database structures between Servers and Clients are identical.
Up to a certain point, the Server can deal with a Client which is of an older version, because all database tables have a ‘minimum version’ number attached, and in addition also an internal version number.
This means that on initial connection, the Server will skip (not add to the Stream) any database tables which it knows the Client cannot deal with. So far so good.
All further data transmissions between Clients and Servers are packed into a small Stream with a standard header, followed by whatever the Client is transmitting.
The Server will copy this Stream to all connected Clients (and in case of a Master Server, to all other connected Local Database Servers).
Any Client or Server which is too old, and does not recognise the Stream, will drop it with an internal error.
Basically, you do not want all this to happen, all Clients and Servers should always be up-to-date.
Unfortunately, because the SARTrack system is still in heavy development, new features (requested by you) are being added all the time, and therefore also new Database tables.
The real problem occurs when a structural change is required with an existing Database table. I try very hard to avoid a situation like that, but sometimes I simply cannot get around it, when new features require that internal changes must be made in existing tables, to make the new feature possible at all.
When this happens, any Server could still deal with a connecting older Client, because it knows its Version number, and could still transmit the older version of the table.
Sadly, this principle does not work when multiple Database Servers are linked together, as is the recommended way of using SARTrack.
Because, when an up-to-date Client transmits a new table format into a Stream to the Server it is connected to, this Server re-transmit this Stream to any other connected Servers, which in turn distribute it to their own range of connected Clients, and they have no control over the contents of the original Stream.
At this point, any Clients (including Local Database Servers) with a too-old Version will fail with a ‘corrupted Stream’ error.
And all this means that in a situation like this, all Database Servers and all SARTrack Clients must be updated to the latest version at the same time.
And this includes the Master Internet Servers from SARTrack Limited itself, the Germany based one and the New Zealand based one!
So I do my very best to avoid this situation.
Recommended practice:
When you start a new Operation or training, always download the latest Server, Client and Forms packages from the SARTrack website, and copy them onto a USB stick. Takes this with you to location, and update any Servers and Clients as required.
Any Internet based Master servers can be updated by clicking on the Server Monitor icon in the Windows Icon Tray, and selecting Check for Updates.
25 November 2017
Bart Kindt
CEO and developer