Rules-based validation in datamaps #15

New issue

Closed

opened 2021-01-28 15:10:48 +00:00 by yulqen · 2 comments

yulqen commented

2021-01-28 15:10:48 +00:00

(Migrated from github.com)

I'm very pleased to see that you're totally revving up to the idea of this stuff, @banillie. It's where the real value of this software is going to come to the fore, I think.

Here are the rules I am tentatively considering at this stage:

NUMBER>X, NUMBER<X
NUMBERANY(X, Y, Z) (X, Y, Z are numbers)
DATE>X, DATE<X
DATERANGE(D1, D2)
NOTEMPTY
TEXT>X; TEXT<X (X is number of chars)
TEXTANY(X, Y, Z) (X, Y, Z are single words)

More than happy to hear your thoughts and ideas.

Originally posted by @yulqen in https://github.com/yulqen/datamaps/issues/14#issuecomment-769150545

I'm very pleased to see that you're totally revving up to the idea of this stuff, @banillie. It's where the real value of this software is going to come to the fore, I think. Here are the rules I am tentatively considering at this stage: * `NUMBER>X, NUMBER<X` * `NUMBERANY(X, Y, Z)` (X, Y, Z are numbers) * `DATE>X, DATE<X` * `DATERANGE(D1, D2)` * `NOTEMPTY` * `TEXT>X; TEXT<X` (X is number of chars) * `TEXTANY(X, Y, Z)` (X, Y, Z are single words) More than happy to hear your thoughts and ideas. _Originally posted by @yulqen in https://github.com/yulqen/datamaps/issues/14#issuecomment-769150545_

yulqen commented

2021-04-02 15:01:13 +00:00

(Migrated from github.com)

Comment from @banillie from now-closed issue:

Hi Matt, I think the above pretty much summarises where we got to on the data validation side of things.

Only thing I can add now is regarding the EMPTY type and whether this should generate a FAIL is the value returned is in fact empty.

I think a good way to handle this is via the specification of NOTEMPTY. The user should have to place this in addition to the actual type of value expects e.g. TEXT. The absence of a return value will then generate a FAIL. However, if notempty is not specified then the assumption is that it's ok for the value to be empty. Maybe at this point the validation can flag empty rather than fail? This will be a useful way for the user to prioritise keys that really need to be returned, from those that are nice to have but aren't going to cause major problems.

Relevant in this issue.

Comment from @banillie from now-closed issue: > Hi Matt, I think the above pretty much summarises where we got to on the data validation side of things. > Only thing I can add now is regarding the EMPTY type and whether this should generate a FAIL is the value returned is in fact empty. > I think a good way to handle this is via the specification of NOTEMPTY. The user should have to place this in addition to the actual type of value expects e.g. TEXT. The absence of a return value will then generate a FAIL. However, if notempty is not specified then the assumption is that it's ok for the value to be empty. Maybe at this point the validation can flag empty rather than fail? This will be a useful way for the user to prioritise keys that really need to be returned, from those that are nice to have but aren't going to cause major problems. Relevant in this issue.

yulqen commented

2021-05-28 10:14:53 +00:00

(Migrated from github.com)

Make the NOTEMPTY validation the priority. NOTEMPTY takes precedence, then tests for other type, such as DATE.