Key Entry App

Version 1.0.1

Rafael Posada

17th October 2017

1 Introduction

The Application Key Entry (keyEntry-App) has been developed under the SASSCAL initiative (http://www.sasscal.org) through the Climate Task Historical and ongoing climate data management to facilitate the key entry of climate data in the meteorological services of Angola, Botswana and Zambia.

The App gives the user a web-based interface to enter the data in the same way as they are in the on-paper form. The user can also custom the structure of the forms and create their own templates. It includes an in-situ quality control based on absolute limits, so that the user will be aware if a key entered value is outside the limits.

The tool has been developed under Shiny, an open source R package that provides a powerful web framework for building web applications using R (https://shiny.rstudio.com/). Shiny helps turn data analyses into interactive web applications without requiring HTML, CSS, or JavaScript knowledge (RStudio, 2017)

This manual will give an overview of the Application, explaining how to install it, how to run it and how to use it.

2 Requirements

2.1 Software requirements

The following software is required for the installation of the App:

Windows 7 or higher
Java SE Runtime Environment 7 or higher (http://www.oracle.com/technetwork/java/javase/downloads/index.html)
R v3.2.1 or higher (https://cran.r-project.org/bin/windows/base/)
RStudio v0.99.467 or higher (https://www.rstudio.com/products/rstudio/download/)
Rtools v.3.2 or higher (https://cran.r-project.org/bin/windows/Rtools/)
A web-browser, such as Mozilla-Firefox, Chrome or Internet Explorer (v.11 or higher)

Besides this software, the App also requires additional programs to be fully operational:

Pandoc (http://pandoc.org/): A universal document converter. It will be required to build up the help files.

If this program is not available in the localhost, the App will install it automatically. The following Table shows the default version that the App will install, as well as the links to the download file. The user can install a later version of the software at any time.

Program	Default version	Link to downloader
Pandoc	v1.19.2.1	http://pandoc.org/installing.html

2.2 R-Packages

The package is the fundamental unit of shareable code in R. A package bundles together code, data, documentation, etc. and is easy to share with others (Wickham, 2017).

The keyEntry-App requires a number of R-packages to run properly. Most of them are available on-line and will be installed automatically by the App. These packages are:

## installr
## jsonlite
## mailR

## rhandsontable
## rmarkdown
## shiny

## shinyBS
## uuid
## XLConnect

The packages are saved in the following path:

## keyEntry_standalone/keyEntry_App/www/R_pkgs/win.binary

3 Install and uninstall

Note: Please, run the installation as Administrator

3.1 Install

The Application is stored as a .zip file to reduce the size of it. There are two different .zip files available:

keyEntry_compact_small.zip (approx. 9 Mb): Contains the minimum information to install the App. The user will require Internet connection to complete the installation, since the App will have to download the R-Packages from a R-CRAN repository (typically the repository: http://cran.us.r-project.org) and the required software (i.e. Pandoc)
keyEntry_compact_large.zip (approx. 49 Mb): Contains all the R-Packages and software required for the installation of the App. The user will not require Internet connection.

To proceed with the installation, the user has to follow these steps:

Unzip file: The user has to unzip the keyEntry_compact_small.zip or keyEntry_compact_large.zip file and place the content in any folder (e.g. Desktop, Documents or any other).
Open keyEntry_standalone folder: This folder contains all the information to install and run the App.
Edit file keyEntry.bat: Right click on that file and select Edit. An editor will open the .bat file and the user has then to modify the path where R is located. The default path is:
```
## C:/Program Files/R/R-3.2.1/bin/i386
```
where R-X.X.X is the version of R, and i386 contains the 32bit version of R. Once the changes have been done, please save the changes and close the Editor.
Run keyEntry.bat: Double click on the file. A command window will pop-up.
Select Meteorological Service: The user will be asked to enter the meteorological service (either DMS, INAMET or ZMD). This information has to be entered only once: during the installation process.

Figure 1. Select the Meteorological Service

Download packages: If the user uses the file keyEntry_compact_small.zip, then the App will download the required R-Packages automatically from an R-CRAN repository. Otherwise, the App will contain already the required packages. The packages will be saved in:
```
## keyEntry_standalone/keyEntry_App/www/R_pkgs/win.binary
```
Note: This step may take some minutes. Please be patient.
Unpack and install libraries: Once the packages are saved locally, the App will proceed to unpack and install the packages as libraries. These libraries will be located in:
```
## keyEntry_standalone/keyEntry_App/www/libraries
```
Note: This step may take some minutes. Please be patient.
Download and install Pandoc: If not available in the keyEntry_standalone folder, the installer file of Pandoc will be downloaded from the Internet. Once the installer file is available, it will be run automatically so that the installation wizard pops up. The user will have to follow the steps of the wizard.

Once the installation process is finished, the keyEntry-App will op en automatically.

The App can be stopped at any time by closing the command window. To re-run the App, execute the file keyEntry_App.bat again. The KeyEntry_App will then pop-up automatically.

3.2 Uninstall

To uninstall the App, just remove the keyEntry_standalone folder.

4 Running the App

To run the application, just double click on the file KeyEntry.bat. After a few seconds, a web browser opens. The KeyEntry-App interface appears directly on the web-browser.

IMPORTANT NOTE

If more than one user will work with the App at the same time, it is recommended to create different sessions, one for each user. To do so, you have to modify one command line in the keyEntry.bat file.

Where it says set users=1, replace the number 1 with the number of users that are going to be connect to the App (see Figure 2).

Each session will have a different port number (30XX) where XX is the user number. For instance, if users is set to 10, then there will be ten ports available (from 3001 to 3010). Maximum number of users is 99.

Figure 2. Set up the number of users that are going to work with the App

Once the user is in the Web-browser, he/she will have to select the form in which the data has to be key entered. After selecting the form, new fields pops up to complete the entry of additional metadata, such as station id, date or name of the person that is entering the data. Once all the fields are filled in, a Create button appears.

Once the user press Create, two tabs appear:

Key Entry Form: Containing the actual key entry form
Limits & Scale Factors: Containing the minimum and maximum limits accepted by the App, as well as the scale factor of each element

In the following sections both tabs will be explained more in detail.

Note: The keyEntry-App can also be accessed from other computers connected to the Intranet. To access to the App from another PC, open a browser (e.g. Mozilla) and type the following IP-address:

## http://172.21.254.162:30XX

where 30XX is the port number and XX refers to the user number.

5 Key Entry

The Key Entry tab contains a table in which the user can actually enter the data (see Figure 3). The structure of this table is defined previously in an Excel file, but it is based on the structure of the original on-paper form (see Section ‘Form Template’ for more details).

Figure 3. Example of a Key Entry Form

5.1 Save the data

The data entered are saved automatically everytime the user enters a new value. This ensures that the data will not be lost if, e.g. the user closes the browser accidentally, or if the computer shuts down unexpectedly.

Alternatively, the user can save the data manually by pressing the button Save data. Then an Excel File (.xls) with the data will be created and saved in the following path:

## keyEntry_standalone/keyEntry_App/www/tmp_files

The information contained in the .xls file is splitted in four Worksheets:

metadata: containing the information related to the metadata, such as station id, form name, contact person or date when file was created
**data:** containing the data entered by the user
limits: containing the limits applied to the form and the scale factor of each element that is available in the form
climsoft: containing information related to CLIMSOFT (i.e. element abbreviation, units, time period, element code, etc.). This worksheet is essentiasl to later import the data into CLIMSOFT successfully

Note: It is recommended that the user presses the Save data button after one form is completed.

5.2 Download the data

To download the data entered, the user has two options:

Download as .csv: The user will get the data in a comma separated file. It will only get the data. Metadata and limits are ignored.
Download as .xls: The user will get the metadata, data and limits in an Excel file. The structure of this file is the same as the one described above (see ‘Save data’)

5.3 Send the data per E-Mail

The user can also send the data per E-Mail. If the user selects this option by pressing the button Send data per E-Mail, a new window will pop-up. The new window expects the user to select one E-Mail account from which the data can be sent, and another E-Mail account to which send the E-mail. The password of the sender is necessary to send the data successfully.

The E-Mail will be then generated automatically and will contain the data in two forms: as a .csv file and as a .xls file.

Note: The E-Mail address of the sender has to be a GMAIL-Account. This account has to be set up to allow ‘less secure apps to access the account’ (see https://support.google.com/accounts/answer/6010255?hl=en for more details)

6 Edit limits and scale factors

6.1 Limits

This tab shows the maximum and minimum values that are accepted when entering data. The limits are defined for each element independently and can be modified by the user (see Figure 3). To do so, it is enough to change the values of the maxValue and minValue.

To make the changes effective, the user will have to press the button Create. This changes are only applied to the current form. This means that new forms will remain with the default limits. If the user wants to modify the limits for every form, then the changes should be done in the form template (see Section ‘Form Template’ for more details).

If the user enters a value beyond the limits defined in the Edit limits tab, then the cell will turn red. In this way the user will be aware that the value does not satisfy the absolute limit control. It is the user the one that has to decide whether the value is actually correct or not. If not, the corrected value should be entered. If it is correct, it is recommended that the user adapts the limits to that value.

6.2 Scale factors

The scale factor refers to a multiplicator that has to be applied to the entered value so that you get a correct end observed value.

Example: it is usual to enter temperature values without decimal separators. This means, that the user enters a temperature of 25.7 C as 257. To retrieve the correct observed value (25.7) we would need to multiply the entered value by 0.1. This 0.1 value is the scale factor of temperature.

Figure 4. Example of Limits and scale factors

7 Excel form template

The structure of the forms has been developed based on the original on-paper documents, so that they look as similar as possible to the original forms. This structure is defined in an Excel file (.xls) which name matches the name of the original Form (e.g. MOZ304A.xlsx contains the structure information of the form MOZ304A of ZMD).

Each .xls file includes the description of the elements to be entered in the form, as well as the main inputs needed to create the form, such as station_id or date. This information is stored in Worksheets, each of which is described in the following.

7.1 Inputs Worksheet

inputs is here understood as the metadata that define the key entry form, such as station id, station name, date or person that complements the Form. These inputs are actually the fields the user has to fill in before a new Form can be created (see Figure 5).

Figure 5. Example of inputs in the App (MOZ304A, ZMD)

The inputs may vary from one form to another, and therefore the user has to be able to modify or create their own inputs based on their own on-paper forms.

To facilitate this, the .xls file that defines the structure of a given form (e.g. MOZ304A, ZMD) includes a Worksheet named inputs in which the input fields are described (Figure 6).

Figure 6. Example of inputs Worksheet (MOZ304A, ZMD)

This Worksheet contains the following information:

code*: Identifier of the inputs.
input*: Name of the inputs. This name will be used to label the field box in the keyEntry-App (see Figure 4).
description: Brief description of the input.
character_limits: How many characters should have the input.
width*: Width of the field box to appear in the KeyEntry App, in pixels.

Note: The fields marked with * are mandatory.

The user can add as many inputs as desired, but it is recommended to enter only those that are required for the on-paper form, so that the form remains similar to the original on-paper document.

To avoid mistakes when typing text in these fields, the user can also define the accepted values for each input in a separate Worksheet. For instance, if one of the input fields is year, then the user can create a Worksheet called year in which the years are defined. Then the user will have to select one year instead of typing the date him-/herself. This should avoid inconsistencies in the date format.

Note: It is mandatory that the user calls the Worksheet with the same name as the input, so that the App can recognize the Worksheet as additional input information. Otherwise the App will ignore the Worksheet

7.2 Elements worksheet

Similarly as with the inputs, the elements that appear in the key entry form should be defined in the .xls file. There is a Worksheet called elements that should contain all the information related to the elements. Figure 7 shows an example of the elements Worksheet. These elements are those contained in the MOZ304A form.

Figure 7. Example of elements Worksheet (MOZ304A)

This Worksheet contains the following information:

element_type: Type of element (e.g. temperature, pressure, windspeed, etc.).
element_abbr*: Abbreviation of the element. This column will be used as headers for the form. It is recommended that it contains the same names as those used in the on-paper form
element_name: Complete name of the Elements
description: Description of the Element
climsoft_code: Element code used in Climsoft to identify the element. This code can be found in the obs_element table of the MS-Access Climsoft database, or in the obsElement table of the mariadb Climsoft database. The climsoft_code will be used to transfer the data from the key-entry Form into a Climsoft db
maximum: Maximum value accepted for a given element. It is used to carry out the absolute limit check when typing the data. If emtpy, the App will not be able to check if the values entered are above this limit
minimum: Minimum value accepted for a given element. It is used to carry out the absolute limit check when typing the data. If emtpy, the App will not be able to check if the values entered are below this limit
units: Units of the element
rule: A numeric value indicating that the element is calculated based on another elements of the form. The number is only an identifier to know which rule has to be applied in order to calculate the value. Please, check the Section Rules definition rules for more details
time_period*: Whether the element is a daily, subdaily, yearly or subyearly value. This information will be usefull for transfering the data into a Climsoft db
time: If the time-period is “subdaily”, then it is required to specify the exact time in which the measurement was done

Note: The fields marked with * are mandatory.

7.3 Rules definition

The rules define how to calculate a value of an element based on other elements of the Form. The calculated values will appear in the Form automatically, and the user cannot overwrite it. This is to ensure that the calculated values cannot be modified manually.

If the calculated values are wrong, then the user will have to check the values of the other elements that have been used to make the calculation.

The rules are defined previously by the user in the form template. Each rule will have its own Worksheet in the .xls file, and the name of that worksheet has to be as follows:

_ruleX_description_

where X is a numeric value indicating the rule_id, and the description a brief description of the element to calculate. For instance, rule1_mean_temp refers to the first rule (Rule Nr. 1), which is defined to calculate the mean temperature.

Currently, there are three different type of rules available:

mean: Calculates the mean value from two elements (Figure 8)
Figure 8. Calculation of Mean Temperature based on the minimum and maximum temperatures
diff: Calculates the difference between two elements (Figure 9)
Figure 8. Calculation of wind speed difference between wind speed at 17 UTC and 06 UTC.
between: Given a value, it will be checked against a table (called between_table) to know whether the value is between a certain range of values. If so, the final calculated value will be a value defined in that table. The between_table should contain the min and max values of each range, and another column with the equivalent value which will be used as calculated value in the end. Figure 10 shows an example of the between rule
Figure 10. Calculation of 24h mean wind speed (24Hr. WIND Mean) based on the wind difference element (24Hr. Wind Diff)

In all cases, the rule Worksheets should contain, at least, the following information:

rule_id: number of the rule. The number has to be unique, and has to match the number used in the element Worksheet in the rule column to define the element of interest.
element_name: Abbreviation of the element that wants to be calculated. It has to match the abbrevation used in the element Worksheet.
formula: Type of formula used to make the calculation (mean, diff, or between)

8 Create new form template

The user can also create a complete new template. It is recommended, however, that the user take advantage of the already existing forms to create their own templates. Therefore, the recommendation is to copy an old form and modify it.

The forms currently available are organized by meteorological service and time period. The form templates currently available are:

	DMS	INAMET	SAWS	ZMD
c(“daily”, “hourly”)
daily	C21.xlsx, MOZ304A.xlsx, diary.xlsx	C21.xlsx, Prova.xlsx	SAWS_daily.xlsx, diary.xlsx	MOZ304A.xlsx

and are located in the folder:

## keyEntry_standalone/keyEntry_App/www/forms

Once the new template has been created, it will be automatically available in the keyEntry_App.

9 References

RStudio (2017). Easy web applications in R. https://www.rstudio.com/products/shiny/ [last accessed: 11.04.2017]

Wickham (2017). R Packages. http://r-pkgs.had.co.nz/intro.html [last accessed: 12.04.2017]

Wikipedia (2017). Open Database Connectivity. https://en.wikipedia.org/wiki/Open_Database_Connectivity [last accessed: 11.04.2017]