7.1.6. Custom data from WFL family loggers

Overview

*I try to keep this topic up-to-date, but be aware that new WFL firmware revisions may add new data export options not yet included here.

*These notes are mostly about custom data export from the WFL logger itself, but will also apply where noted to the Meteo-Pi Ethernet logger.

*This topic covers the main aspects of exporting custom data files from WFL. But be aware that WFL performs extensive processing of the weather station data and may be capable – at least in principle – of additional data export options not explicitly covered here.

WFL family loggers can export data in many different ways. This topic explains the various options.

First, let’s distinguish between data packets requested (ie pulled) by familiar weather station programs such as Weatherlink for Windows, CumulusMX etc versus the customised export of specific data files.

WFL supports the well-known LOOP and Archive data formats used by Davis stations to pass real-time and summary data packets to a requesting program. Since WFL (and Meteo-Pi Ethernet) are network loggers, this Davis-format packet data passes across the network, usually via port 22222, with WFL effectively emulating the legacy Davis WeatherlinkIP loggers. These direct logger connections are usually made across the local network to software running on a local computer, although in principle these transfers could also be configured to happen across the Internet. This present topic does not cover this type of local data-packet connection to a standard weather station program like CumulusMX – such data connections should just work, with any necessary troubleshooting advice being provided elsewhere in this section – but instead focuses on the other custom export of complete data files that WFL can provide (usually, but not necessarily, to servers across the Internet rather than locally).

Now, let’s make one further distinction: WiFi Logger (WFL) can export (or push) data files to two types of server:

• Specific weather platforms in the cloud such as weatherlink.com and Weather Underground etc. In this case, data is packaged in the required file format for the target platform;
• Data packaged in generic file types that can be pointed towards a server of the user’s choice, either local or remote.

For WFL, the specific weather platforms are configured in the Setup | Data Export Options menu, while the generic export options will be managed within the Setup | Custom Export Setup menu. The Meteo-Pi Ethernet (MPE) logger has fewer export options and these are all contained under the Export Setup menu. NB The original Meteo-Pi and DIY loggers are simple interface loggers; these have no user interface and hence no data export options.

Configuration for uploading to specific weather platforms such as weatherlink.com or Weather Underground simply involves entering the appropriate authentication details for each platform and adding any other platform-specific settings. These steps should be fairly obvious to do and again don’t need further comment here.

However, the various options for the generic/custom exports are less obvious and are the main subject of this topic.

Understanding the Custom Data Export options

*Note: These custom data export options are provided for the convenience of more technically minded users. But their use does require understanding from the user and data management on the target server. One important aspect is that the logger cannot anticipate the time and date range of any data files that may be requested and so, especially for multi-record files containing archive data, will export whatever default block of data is configured in the logger settings to send. It is the user’s responsibility to code their program running on the target server so as to select data records appropriately from the overall data file and to minimise gaps and duplication of data that may be stored in databases, for example. Weather station programs such as CumulusMX and Weatherlink for Windows already take care of issues like this, as do cloud platforms like weatherlink.com, Weather Underground etc. But if you are rolling your own software then you will need to manage these date/time aspects yourself. So if using custom datafile exports, please be aware of the need to curate the received data file appropriately at the target server.

*The options listed below are those available from the WiFi Logger. The Meteo-Pi Ethernet Logger currently has only a single custom export option – the one labelled HTTP>>PHP>>MySQL.

WFL provides numerous options for using custom data exports and it can be confusing as to which options should be used for which purpose. However, all the alternatives effectively involve the export of a single data file at a given time point and the alternatives are all variations involving three aspects of the file export, as discussed further below:

• The granularity and frequency of the data within the data file
• The file contents and file format
• The protocol being used to upload the data

Data granularity/frequency

As most users will be aware, Davis weather stations can generate two classes of data: current conditions data (also sometimes called real-time data) and summary or archive data.

Current conditions data is a snapshot of sensor readings at a given timepoint and this data sample is regenerated at frequent intervals, as set in the logger configuration. Data files containing current conditions data will contain a single set of readings from all configured sensors and in a format as specified by selected file type. Davis stations can generate new current-conditions readings as often as every 2.5 seconds, but it is rarely necessary or practicable to push data at this frequency across the Internet. It is more common to use a minimum interval of 1 minute or longer, potentially up to a maximum period of 1-2 hours. The frequency is typically configurable within the WFL settings for each upload type.

Archive data is data summarised by the Davis console over a configurable interval from 1 minute to 2 hours, so a single set of readings will contain mean sensor values over the duration of each archive interval. Data files containing archive data will typically contain multiple archive data records covering up to many days of console/logger operation

These two classes of data really serve distinct purposes, with current conditions data being a single snapshot of the latest sensor readings, while the archive data is a more convenient data source to build into a permanent table or database of weather conditions over periods of days, months, years etc, (NB Current conditions data can also be assembled into a full long-term data record at the target server, but using pre-summarised archive data can be a more accurate and more appropriate solution, for reasons that are really beyond the scope of these brief notes.)

File contents and format

TBA

Upload protocol

Protocol in this context means the formal network/Internet protocol by which file transfer is implemented, for example FTP, or MQTT etc. The same file format can be uploaded by different protocols but obviously the sending configuration must match what the target server expects in terms of protocol, port number, authentication credentials etc.

For Custom data exports, WFL currently supports four distinct protocols, as listed below. Note that these are all push protocols in the sense that the logger initiates the data file transfer (at the user’s command in the case of the manual archive transfer, but automatically to a programmable schedule in the case of the other 3 protocols). This is in contrast to the pull requests for LOOP and archive data packets made to the logger by external software, as mentioned in the overview section above. Note that for push protocols, the server must be awake and listening for the data file when it is sent, if data is not to be lost..

Manual archive:

Automatic legacy FTP:

Automatic MQTT

Automatic Unknown::

This topic is still being written.

All content © Prodata Associates Ltd and www.weatherstations.co.uk
Copying or linking to this knowledge-base is prohibited without permission
Please report unauthorised use to support@weatherstations.co.uk