Import Processing Mechanism (Full-featured)
This manual is in pilot operation.
|
Overview
In the import process, data in the target area is cleared before import data is loaded. That is, the overall import process proceeds as follows:
-
On the import destination screen, clear the values of cells included in the import target area.
-
Load data from the import source file and reflect it in the screen data.
Although the data displayed on the screen is changed in Step 2 above, these changes are not yet saved to the database. To save them, as with normal data modification, you must press the Save button.
Details of the processing in each step are explained below.
Clearing the Import Target Area
In the first step, the values of cells included in the import target area are cleared on the screen generated by the form. The target cells are those that meet the following conditions:
-
Cells that are editable on the import destination screen (not limited to cells currently displayed on the screen; this also includes cells that become visible and editable when items are selected in selection boxes on the vertical axis).
-
Cells of the "Source ledger" specified in the Source Ledger field of the Import Specification. If no target is specified in the Source Ledger field, all ledger cells that are editable in the form are targeted.
Loading Data from the Import Source File
In the second step, based on one row of data loaded from the import source file, code translation and code concatenation are performed. Then, values are written to (or skipped for) ledger cells keyed by the values of fields (translated values) that are mapped by label to the dimensions used in the import target ledger. This process is repeated for all rows in the import source file. However, if the import source file contains rows at the beginning that should not be imported, you can specify the number of such rows in "Number of Header Rows to Skip" in the Import Specification to skip them.
If multiple Value Fields are specified in the Import Specification, the above process is repeated for each row and for each Value Field (for details on how to configure writing the values of each Value Field to separate ledger cells, refer to "Considerations When Multiple Value Fields Exist" described later).
The data loading process is executed in the order of rows and in the order of Value Fields within each row. If a ledger cell has a numeric attribute, values written to the same ledger cell are accumulated. If it is a non-numeric cell, the last written value becomes effective.
Below, the processing for one row and one Value Field is explained using an example.
Example
(1) Layout and Contents of Import Data
Assume that monthly expense amounts by business unit, department and account are imported. The Key Fields consist of Fiscal Year, Month, Business Unit, Department and Account, followed by the Value Field "Amount". The data used in the explanation is shown below:
(Fiscal Year) (Month) (Business Unit) (Department) (Account) (Amount)
2016, 05, 10, 111-22, 12345, 1000
⚠ In the explanation below, "Amount" is assumed not to be defined as a Data source field. That is, Amount is treated as an "Implicit Value Field". Even if Amount is explicitly defined as a Value Field, the essence of the explanation below does not change (the Amount field simply becomes one of the Data source fields).
(2) Import Destination Ledger
The label of the import destination ledger is "EXPENSE". For each dimension that composes this ledger, the member label setting requirements at import are as follows:
- Accounts (
#ACCOUNT) -
The "Account" value ("12345") from the import data is used directly as the member label.
- Changes (
#CHANGE) -
A fixed value "#NONE" is set regardless of the code in the import data.
- Fiscal Year (
#FY) -
The "Fiscal Year" value ("2008", etc.) from the import data is used directly as the member label.
- Relative Period
-
Corresponds to the "Month" in the import data, but conversion is required, such as "04" to "M1", "05" to "M2".
- Scenario
-
A fixed value "ACTUAL" is set regardless of the code in the import data.
- Views (
#VIEW) -
A fixed value "PER" is set regardless of the code in the import data.
- Division/Department (
DIV_DEPT) -
Concatenate the first three digits of "Department" to "Business Unit" and prefix with "D" (for example, "D10111").
Explanation of Processing Based on the Example
Step 1. Setting Values to Data source fields
When one import data record is read, first, the values of each Key Field from the import data are set to the "Data source fields" defined in the Import Specification. Each field has two areas: "Pre-translation Value" and "Post-translation Value". At this stage, values are set to the "Pre-translation Value".
Regarding how to assign Field Labels:
For fields whose "Post-translation Value" directly becomes the member label of a dimension used in the target ledger, set the Field Label to the label of that dimension. In this example, Fiscal Year (#FY), Relative Period (#PERIOD) and Accounts (#ACCOUNT) fall into this category.
For other fields, assign Field Labels that are different from Dimension Labels. In this example, Business Unit (DIV) and Department (DEPT) are such fields.
Step 2. Code Translation (in Data source fields)
In the next step, Translation Tables are applied to the "Pre-translation Values" of each Data source field, and "Post-translation Values" are obtained. For fields for which no Translation Table is specified, the Pre-translation Value is used as-is as the Post-translation Value.
In this example, Translation Tables TBL_PER and TBL_DEPT are applied to the fields #PERIOD and DEPT, respectively. Their contents are as follows:
- Period Translation Table (
TBL_PER) -
Registers 12 pairs of month and period labels, such as 04→M1, 05→M2, …, 01→M10 ("04→M1" means a row where the pre-translation code is "04" and the post-translation code is "M1").
- Department Translation Table (
TBL_DEPT) -
Contains only one translation rule. The pre-translation code is "???*" and the post-translation code is "???". This extracts the first three digits of the pre-translation code. Refer also to "Wildcards" and "Examples of Translation Rule Specification" in the Translation Table section.
Step 3. Calculating Pre-translation Values of Derived fields
Up to Step 2, member labels for the dimensions #SCENARIO, #VIEW, #CHANGE and DIV_DEPT are not yet determined. Therefore, in the Import Specification, Derived fields with these Dimension Labels are defined so that member labels can be determined.
By setting appropriate expressions in the "Expression" field of each field, the "Pre-translation Value" of each field is calculated according to the expression.
The Pre-translation Values of Derived fields can be set as fixed values as shown for #SCENARIO, #VIEW and #CHANGE above, or calculated from the values of other fields as in the DIV_DEPT field.
The expression DIV & DEPT specified for the DIV_DEPT field in the figure above means concatenating the Post-translation Values of the DIV and DEPT fields. Note that expressions always reference Post-translation Values.
Step 4. Code Translation (in Derived fields)
Next, Translation Tables are applied to the Pre-translation Values of Derived fields to obtain Post-translation Values. For fields for which no Translation Table is specified, the Pre-translation Value is used as-is as the Post-translation Value. Except that the target is Derived fields, this processing is exactly the same as in Step 2.
In this example, the Translation Table TBL_DIVDEP is applied to the field DIV_DEPT. The contents of this Translation Table are as follows:
-
Division/Department Translation Table (
TBL_DIVDEP) … Contains only one translation rule. The pre-translation code is*, and the post-translation code isD*. This prefixesDto the pre-translation code.
In this example, there are no Derived fields that reference other Derived fields in their expressions, so at this point, the "Post-translation Value" fields for all fields are filled. If there are Derived fields that reference other Derived fields, Step 3 and Step 4 are repeated until the "Post-translation Value" fields for all fields are filled. Note that expressions that cause circular references among Derived fields are not allowed and are checked as errors when the form is registered.
Step 5. Writing to Cells
By Step 4, the "Post-translation Value" fields for all fields are filled. As the final step, treating the "Post-translation Values" of fields that have the same labels as the dimensions of the target ledger as member labels, the cell values of the target ledger (specified in the Import Specification) are updated with the Post-translation Values of the Value Fields.
If there are multiple import data records for the same member label combination, amounts and numbers are accumulated, while non-numeric cells are overwritten with the last value.
Import Processing Log
The import processing described above is output to a log file as a processing log. The file name is automatically assigned. The processing log contains Pre-translation Values and Post-translation Values of all fields, amounts and messages indicating the details of exception handling described below.
For details, refer to the document "Import Processing Log".
Exception Handling
Exception handling includes intended exception handling and unintended exception handling (errors). In both cases, data is not written to cells.
(1) Intended Exception Handling
If the calculation result of an Expression or the translation result from a Translation Table becomes the value #IGNORE (string), the data of that Value Field in that row is skipped and excluded from writing to the ledger. Refer also to the Translation Table description.
(2) Unintended Exception Handling (Error Handling)
This includes cases such as code translation errors, field values with the same labels as the dimensions of the target ledger being inappropriate as member labels and data outside the import range being included without being skipped.
Of the two types of exception handling above, "Intended Exception Handling" takes precedence. That is, rows skipped by Intended Exception Handling do not result in errors even if they also meet conditions for unintended exception handling such as code translation errors.
In all cases, messages and a code representing the severity of the exception (#SEVERITY) are displayed in the import processing log. In case (1), the value of #SEVERITY is INFO. In case (2), depending on the severity, it is WARNING or ERROR.
Considerations When Multiple Value Fields Exist
When importing data that contains multiple Value Fields, you typically want to write the values of those Value Fields to separate ledger cells rather than accumulating all of them into a single ledger cell.
-
Define Value Fields by month and write the value of each field to a ledger cell keyed by a different month (Relative Period).
-
Define two Value Fields, such as sales quantity and sales amount, and write the value of each field to ledger cells keyed by the corresponding Accounts.
To support such requirements, a mechanism is provided to specify "Value Field Keys" for Value Fields. For example, when defining monthly Value Fields, set Relative Period labels such as M1, M2, … in the "Value Field Key" field of each Value Field. At the same time, define a Derived field with the same label as the Relative Period dimension (#PERIOD) and specify @field_key in its Expression field.
The @field_key function retrieves the Value Field Key of the Value Field currently being processed. As described earlier, the data loading process is executed in row order and in Value Field order within each row, and Expression evaluation is performed each time. Therefore, when processing the first Value Field (with Value Field Key M1 specified), @field_key returns "M1", and when processing the second Value Field, it returns "M2".
The same approach can be used for importing sales quantity and sales amount. Specify Account Labels to distinguish sales quantity and sales amount in the "Value Field Key" field of each respective Value Field, and specify @field_key in the Expression field of a Derived field with the same label as the Accounts dimension (#ACCOUNT).
Multiple Value Field Keys can also be specified for each Value Field. For example, when importing data that holds monthly budgets and forecasts side by side, specify "BUDGET, M1" and "FORECAST, M2" in the Value Field Key field of each amount column. Meanwhile, if you specify @field_key(1) in the Expression field of the Scenario Derived field, the first comma-separated item, that is, BUDGET or FORECAST, is set. Similarly, by specifying @field_key(2) in the Expression field of the Relative Period Derived field, the second Value Field Key, that is, "M1" or "M2", is set. Note that @field_key and @field_key() without arguments are abbreviations of @field_key(1).
Specifying Translation Tables for Value Fields
It is also possible to specify Translation Tables for Value Fields. While code translation has no meaning for amounts or quantities (if processing is required, Expressions can be used), when importing code values into enumerated Accounts, Translation Tables can be used to import the translated values as the values of the corresponding enumerated Accounts.