Testing version v1.1.2

banillie commented

2021-01-27 12:44:40 +00:00

(Migrated from github.com)

Hi Matt,
using this as a place to log stuff that comes up / I think of while testing.
It all worked really well no crashes during testing.
for datamaps import templates -m

might be useful for there to be an INFO message (in bash) to state validation report has been created.
related to the above some handling of user entering 'type' column into dm.csv. e.g. I had it as 'Type' and therefore datamaps ignored it and didn't produce a val report. I tried datamaps import templates -v and it told me there was no 'type' column. very useful but you could have this messaging in the -m command also.
(expected = Empty and got = Empty ) = Fail.
(expected = " " i.e. a blank and got = Empty) = Fail
is that how you expect the validation to behave?
related to above. Presumably the ability to designate data values as priority/must fill will deal with this issue. i.e. you can specify which data is required, but grade it on how necessary it is for it to be returned.
There are/will be instances where the expected data changes depending on whether another bit of data has been reported. e.g. if a milestone name is reported, then a milestone date needs to be provided. But, if no milestone name is reported then no date is required.

both -z and --inputdir options work brilliant well and are super useful. Especially the zip one, which obvs reduces time moving files around, as well as deleting/tidy afterwards.

Look forward to discuss more tmrw Matt. Cheers, Will.

Hi Matt, using this as a place to log stuff that comes up / I think of while testing. It all worked really well no crashes during testing. for `datamaps import templates -m` - might be useful for there to be an INFO message (in bash) to state validation report has been created. - related to the above some handling of user entering 'type' column into dm.csv. e.g. I had it as 'Type' and therefore datamaps ignored it and didn't produce a val report. I tried `datamaps import templates -v` and it told me there was no 'type' column. very useful but you could have this messaging in the -m command also. - (expected = Empty and got = Empty ) = Fail. - (expected = " " i.e. a blank and got = Empty) = Fail - is that how you expect the validation to behave? - related to above. Presumably the ability to designate data values as priority/must fill will deal with this issue. i.e. you can specify which data is required, but grade it on how necessary it is for it to be returned. - There are/will be instances where the expected data changes depending on whether another bit of data has been reported. e.g. if a milestone name is reported, then a milestone date needs to be provided. But, if no milestone name is reported then no date is required. both -z and --inputdir options work brilliant well and are super useful. Especially the zip one, which obvs reduces time moving files around, as well as deleting/tidy afterwards. Look forward to discuss more tmrw Matt. Cheers, Will.

yulqen commented

2021-01-27 14:01:52 +00:00

(Migrated from github.com)

Hi Will,

I'm INCREDIBLY grateful that you've done testing to this level of detail. It's incredibly helpful for me to have that feedback, so thanks.

I'll address each of your points in the same order:

might be useful for there to be an INFO message (in bash) to state validation report has been created.
** It should already do that at the bottom of the log output
related to the above some handling of user entering 'type' column into dm.csv. e.g. I had it as 'Type' and therefore datamaps ignored it and didn't produce a val report. I tried datamaps import templates -v and it told me there was no 'type' column. very useful but you could have this messaging in the -m command also.
**I will add "Type" to the list of headers that it accepts. At the moment, you can use actually use "type", "value_type", "cell_type" or "celltype" - but not "Type"! (github.com/yulqen/bcompiler-engine@414a172992/engine/utils/extraction.py (L384)). Take your point about warning the user early about what's going on.
(expected = Empty and got = Empty ) = Fail.
(expected = " " i.e. a blank and got = Empty) = Fail
is that how you expect the validation to behave?
** My thinking is that if a cell has a line in the datamap, then it is required. Therefore if the cell is empty, it flags as Fail. I know you currently expect many of the cells to be empty and it's ok for this to happen, so not technically a type-fail, but that's where we are at the moment. I have been thinking about whether we need an Empty type. What is the point of putting something in the datamap if you actually expect it to be empty? Maybe there is a use. Happy to discuss.
related to above. Presumably the ability to designate data values as priority/must fill will deal with this issue. i.e. you can specify which data is required, but grade it on how necessary it is for it to be returned.
** Yes, absolutely right. That should be factored into the discussion about Empty as mentioned above.
There are/will be instances where the expected data changes depending on whether another bit of data has been reported. e.g. if a milestone name is reported, then a milestone date needs to be provided. But, if no milestone name is reported then no date is required.
** Great, that sounds like an awesome bit of functionality and exactly the kind of direction I want to take this with the rules-based type validation

Hi Will, I'm INCREDIBLY grateful that you've done testing to this level of detail. It's incredibly helpful for me to have that feedback, so thanks. I'll address each of your points in the same order: * might be useful for there to be an INFO message (in bash) to state validation report has been created. ** **It should already do that at the bottom of the log output** * related to the above some handling of user entering 'type' column into dm.csv. e.g. I had it as 'Type' and therefore datamaps ignored it and didn't produce a val report. I tried datamaps import templates -v and it told me there was no 'type' column. very useful but you could have this messaging in the -m command also. ****I will add "Type" to the list of headers that it accepts. At the moment, you can use actually use "type", "value_type", "cell_type" or "celltype" - but not "Type"!** (https://github.com/yulqen/bcompiler-engine/blob/414a1729928942d2bdde48e5cb743c4a80b687b1/engine/utils/extraction.py#L384). Take your point about warning the user early about what's going on. * (expected = Empty and got = Empty ) = Fail. * (expected = " " i.e. a blank and got = Empty) = Fail * is that how you expect the validation to behave? ** **My thinking is that if a cell has a line in the datamap, then it is required. Therefore if the cell is empty, it flags as `Fail`. I know you currently expect many of the cells to be empty and it's ok for this to happen, so not technically a type-fail, but that's where we are at the moment. I have been thinking about whether we need an `Empty` type. What is the point of putting something in the datamap if you actually expect it to be empty? Maybe there is a use. Happy to discuss.** * related to above. Presumably the ability to designate data values as priority/must fill will deal with this issue. i.e. you can specify which data is required, but grade it on how necessary it is for it to be returned. ** **Yes, absolutely right. That should be factored into the discussion about `Empty` as mentioned above.** * There are/will be instances where the expected data changes depending on whether another bit of data has been reported. e.g. if a milestone name is reported, then a milestone date needs to be provided. But, if no milestone name is reported then no date is required. ** **Great, that sounds like an awesome bit of functionality and exactly the kind of direction I want to take this with the rules-based type validation**

👍 1

banillie commented

2021-01-27 14:32:02 +00:00

(Migrated from github.com)

Least I can do Matt!

👍 1

banillie commented

2021-01-28 13:53:25 +00:00

(Migrated from github.com)

Just while I think about it another type based rule that would v helpful as we come around all the time in the data is the size/denomination of the data e.g NUMBER can't be larger than 100,000,000

👍 1

yulqen commented

2021-01-28 15:10:23 +00:00

(Migrated from github.com)

I'm very pleased to see that you're totally revving up to the idea of this stuff, @banillie. It's where the real value of this software is going to come to the fore, I think.

Here are the rules I am tentatively considering at this stage:

NUMBER>X, NUMBER<X
NUMBERANY(X, Y, Z) (X, Y, Z are numbers)
DATE>X, DATE<X
DATERANGE(D1, D2)
NOTEMPTY
TEXT>X; TEXT<X (X is number of chars)
TEXTANY(X, Y, Z) (X, Y, Z are single words)

More than happy to hear your thoughts and ideas.

I'm very pleased to see that you're totally revving up to the idea of this stuff, @banillie. It's where the real value of this software is going to come to the fore, I think. Here are the rules I am tentatively considering at this stage: * `NUMBER>X, NUMBER<X` * `NUMBERANY(X, Y, Z)` (X, Y, Z are numbers) * `DATE>X, DATE<X` * `DATERANGE(D1, D2)` * `NOTEMPTY` * `TEXT>X; TEXT<X` (X is number of chars) * `TEXTANY(X, Y, Z)` (X, Y, Z are single words) More than happy to hear your thoughts and ideas.

banillie commented

2021-01-28 21:13:45 +00:00

(Migrated from github.com)

Yes I really agree that it's going to be very useful and powerful. Every time I work on data outputs for reporting, there as always some data missing or reported incorrectly. Sorting / filtering / cleaning / killing these quickly is going to be so useful. All the above look great 👍
look forward to discussing further.

Yes I really agree that it's going to be very useful and powerful. Every time I work on data outputs for reporting, there as _always_ some data missing or reported incorrectly. Sorting / filtering / cleaning / killing these quickly is going to be so useful. All the above look great :+1: look forward to discussing further.

banillie commented

2021-03-15 16:08:08 +00:00

(Migrated from github.com)

Hi Matt, I think the above pretty much summarises where we got to on the data validation side of things.

Only thing I can add now is regarding the EMPTY type and whether this should generate a FAIL is the value returned is in fact empty.

I think a good way to handle this is via the specification of NOTEMPTY. The user should have to place this in addition to the actual type of value expects e.g. TEXT. The absence of a return value will then generate a FAIL. However, if notempty is not specified then the assumption is that it's ok for the value to be empty. Maybe at this point the validation can flag empty rather than fail? This will be a useful way for the user to prioritise keys that really need to be returned, from those that are nice to have but aren't going to cause major problems.

Hi Matt, I think the above pretty much summarises where we got to on the data validation side of things. Only thing I can add now is regarding the `EMPTY` type and whether this should generate a `FAIL` is the value returned is in fact empty. I think a good way to handle this is via the specification of `NOTEMPTY`. The user should have to place this in addition to the actual type of value expects e.g. `TEXT`. The absence of a return value will then generate a `FAIL`. However, if notempty is not specified then the assumption is that it's ok for the value to be empty. Maybe at this point the validation can flag empty rather than fail? This will be a useful way for the user to prioritise keys that really need to be returned, from those that are nice to have but aren't going to cause major problems.

banillie commented

2021-03-15 16:26:27 +00:00

(Migrated from github.com)

also minor thing I think you need to add an explanation for the -z option to the cli --help text output for datamaps import templates. I couldn't see it in there. Looking back at the convo we had above reminded me that it's a really useful option, but I can't remember what the command is.

also minor thing I think you need to add an explanation for the -z option to the cli --help text output for `datamaps import templates`. I couldn't see it in there. Looking back at the convo we had above reminded me that it's a really useful option, but I can't remember what the command is.

yulqen commented

2021-03-16 18:54:26 +00:00

(Migrated from github.com)

It's already in v.1.1.2. Do datamaps import templates --help. Point datamaps at a zip file of populated templates!

It's already in `v.1.1.2`. Do `datamaps import templates --help`. Point datamaps at a zip file of populated templates!

banillie commented

2021-03-17 12:43:04 +00:00

(Migrated from github.com)

Yeap there it is. Don't know how it missed it before. Anyway great feature that I'll show the team.

Rows
Columns

Testing version v1.1.2 #14