Datamap sheet field issues #14

New issue

Closed

opened 2019-10-14 13:02:41 +00:00 by yulqen · 2 comments

yulqen commented

2019-10-14 13:02:41 +00:00

(Migrated from github.com)

If a sheet-block (however many rows pertaining to a particular sheet) of rows in the datamap has its sheet field missing/blank, then that row will now migrate during the export process, However, if the first row of the block is intact and the other sheet fields are missing for that particular block, then the data will actually be migrated.

This will cause a problem if the gap in the sheet entries spans the boundaries between blocks because the program won't know where the change in sheet happens.

Example:

cell_key,template_sheet,cellreference,type
Project/Programme Name,,C11,TEXT  # missing sheet herel the rest of the `Introduction` sheets will migrate
Department ,Introduction,C9,TEXT
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , Introduction, c35,TEXT
String Key,Summary,B3,TEXT
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

cell_key,template_sheet,cellreference,type
Project/Programme Name,Introduction,C11,TEXT
Department ,,C9,TEXT  # second row sheet missing - all `Introduction` sheets will export
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , Introduction, c35,TEXT
String Key,Summary,B3,TEXT
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

cell_key,template_sheet,cellreference,type
Project/Programme Name,Introduction,C11,TEXT
Department ,Introduction,C9,TEXT
Delivery Body ,Introduction,C10,TEXT
GMPP - IPA ID Number,Introduction,C12,TEXT
Controls Project ID number,Introduction,C13,NUMBER
Project Type (for IPA use),Introduction,C14,TEXT
Classification,Introduction,C15,TEXT
GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT
Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE
Snapshot Date,Introduction,C18,DATE
Senior Responsible Owner (SRO),Introduction,C21,TEXT
Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT
Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER
SRO Sign-Off,Introduction,C24,DATE
Bad Spacing , , c35,TEXT # missing last `Introduction` and..
String Key,,B3,TEXT #.. first summary. This row likely end up targeting `Introduction` sheet
Big Float,Another Sheet,F17,NUMBER
Funny Date,Another Sheet,H39,DATE

The code handling this is currently at engine/repository/templates.py(39), _populate_workbook(). The test is test_populate_templates_from_master.py.test_export_continues_with_missing_sheet_in_datamap`.

The best solution here is to use the initial datamap check to ensure that every datamap row has the requisite fields, and warn the user before it procedes.

If a sheet-block (however many rows pertaining to a particular sheet) of rows in the datamap has its sheet field missing/blank, then that row will now migrate during the `export` process, However, if the first row of the block is intact and the other sheet fields are missing for that particular block, then the data will actually be migrated. This will cause a problem if the gap in the sheet entries spans the boundaries between blocks because the program won't know where the change in sheet happens. Example: ```csv cell_key,template_sheet,cellreference,type Project/Programme Name,,C11,TEXT # missing sheet herel the rest of the `Introduction` sheets will migrate Department ,Introduction,C9,TEXT Delivery Body ,Introduction,C10,TEXT GMPP - IPA ID Number,Introduction,C12,TEXT Controls Project ID number,Introduction,C13,NUMBER Project Type (for IPA use),Introduction,C14,TEXT Classification,Introduction,C15,TEXT GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE Snapshot Date,Introduction,C18,DATE Senior Responsible Owner (SRO),Introduction,C21,TEXT Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER SRO Sign-Off,Introduction,C24,DATE Bad Spacing , Introduction, c35,TEXT String Key,Summary,B3,TEXT Big Float,Another Sheet,F17,NUMBER Funny Date,Another Sheet,H39,DATE ``` ```csv cell_key,template_sheet,cellreference,type Project/Programme Name,Introduction,C11,TEXT Department ,,C9,TEXT # second row sheet missing - all `Introduction` sheets will export Delivery Body ,Introduction,C10,TEXT GMPP - IPA ID Number,Introduction,C12,TEXT Controls Project ID number,Introduction,C13,NUMBER Project Type (for IPA use),Introduction,C14,TEXT Classification,Introduction,C15,TEXT GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE Snapshot Date,Introduction,C18,DATE Senior Responsible Owner (SRO),Introduction,C21,TEXT Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER SRO Sign-Off,Introduction,C24,DATE Bad Spacing , Introduction, c35,TEXT String Key,Summary,B3,TEXT Big Float,Another Sheet,F17,NUMBER Funny Date,Another Sheet,H39,DATE ``` ```csv cell_key,template_sheet,cellreference,type Project/Programme Name,Introduction,C11,TEXT Department ,Introduction,C9,TEXT Delivery Body ,Introduction,C10,TEXT GMPP - IPA ID Number,Introduction,C12,TEXT Controls Project ID number,Introduction,C13,NUMBER Project Type (for IPA use),Introduction,C14,TEXT Classification,Introduction,C15,TEXT GMPP (GMPP - formally joined GMPP),Introduction,C16,TEXT Reporting period (GMPP - Snapshot Date),Introduction,C17,DATE Snapshot Date,Introduction,C18,DATE Senior Responsible Owner (SRO),Introduction,C21,TEXT Senior Responsible Owner (SRO) - Email,Introduction,C22,TEXT Senior Responsible Owner (SRO) - ID number,Introduction,C23,NUMBER SRO Sign-Off,Introduction,C24,DATE Bad Spacing , , c35,TEXT # missing last `Introduction` and.. String Key,,B3,TEXT #.. first summary. This row likely end up targeting `Introduction` sheet Big Float,Another Sheet,F17,NUMBER Funny Date,Another Sheet,H39,DATE ``` The code handling this is currently at `engine/repository/templates.py(39), `_populate_workbook()`. The test is `test_populate_templates_from_master.py.test_export_continues_with_missing_sheet_in_datamap`. The best solution here is to use the initial datamap check to ensure that every datamap row has the requisite fields, and warn the user before it procedes.

yulqen commented

2019-10-25 15:32:52 +00:00

(Migrated from github.com)

New test required in test_datamap_parser.py - stub created. Need to focus on engine.utils.extraction.datamap_reader() which should reject any badly formatted datamap file. All exceptions raised here should be passed to main() and handled with appropriate log messages.

New test required in `test_datamap_parser.py` - stub created. Need to focus on `engine.utils.extraction.datamap_reader()` which should reject any badly formatted datamap file. All exceptions raised here should be passed to `main()` and handled with appropriate log messages.

yulqen commented

2019-10-26 15:08:01 +00:00

(Migrated from github.com)

Fixed by raising exception when sheet field is missing, and datamaps quits, asking the user to correct the datamap.

Fixed by raising exception when `sheet` field is missing, and datamaps quits, asking the user to correct the datamap.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

yulqen/bcompiler-engine#14

No description provided.

Rows
Columns