Walls: Overly aggressive warnings when using partial name for units of measurement in #UNITS A|V|D|S etc

[Joe Walko]

Cavern should not warn on undocumented usage of shorthand units of measure in #UNITS. #UNITS A=DEG is commonly used; the latest build now warns on this.

Walls only cares about the first character when specifying units of measurements with #UNITS A|V|D|AB|VB|S, etc.

For example, with #UNITS A=DEGREE, Walls only cares about the "D". Thus, for better or for worse, these are all valid in Walls as argument to #UNITS A: A=Degree, A=deg, A=De, A=Dxyzzy, A=DegreesWhatever

A reasonable compromise would be to warn when, the argument is not legitimate shorthand. For example, "DEG", "DE", DEGR" would be OK, but "Dxyzzy" would not (even though Walls accepts Dxyzzy as shorthand for degrees).

Note Walls does not extend this same flexibility when specifying units as a suffix - single character is enforced in those cases.

[Olly Betts]

It's too late to change this for 1.4.23 I'm afraid.

The Walls documentation says you can abbreviate to just the capitalised part (so e.g. Degrees means it can be abbreviated to D). Deviations from this such as #seg for #Segment as well as #s are explicitly noted. So DEG is not documented as allowed currently - it'd be really good to resolve this in the Walls documentation and/or code.

Andy has indicated he was planning to warn about such cases in his new parser, so it seemed useful for us to do so too.

A reasonable compromise would be to warn when, the argument is not legitimate shorthand. For example, "DEG", "DE", DEGR" would be OK, but "Dxyzzy" would not (even though Walls accepts Dxyzzy as shorthand for degrees).

That seems a reasonable rule, and fairly easy to implement.

Are you actually seeing anything other than DEG? Another option would be to just hand-pick a few acceptable abbreviations. None of the other units seem to naturally abbreviate to anything other than the first letter, so beyond DEG perhaps allowing leaving off the S (so METER, DEGREE, GRAD, MIL) and maybe allowing METRES/METRE (the correct spelling in the UK and elsewhere).

If we go with allowing any truncation I'd suggest this only apply to units, and not to e.g. TYPEAB=NO TYPEVB=CORRECT - I've never seen those written as anything other than N and C in real data.

I ran the latest code against all the public datasets I've found, which suggested this warning wasn't going to be problematic. (All were clean except for a handful of A=Deg in the Thailand one, but that's a messy dataset in general and fixing a handful of occurrences to eliminate use of an undocumented feature seemed good.)

This highlights that I can't test on probably the largest and most complex Walls dataset. I realise it may not be politically feasible, but it seems worth asking if it's possible to allow me access on a non-disclosure basis. It'd definitely help with working on Walls support.

[Joe Walko]

"DEG" is my issue. About 5,000 of them :-)

From a practical standpoint, accepting "DEG" without warning would resolve the issue - I'm not seeing other variants. "DEG" is a not uncommon informal abbreviation of "DEGREE", hence silently accepting "DEG" would be pragmatic. Even the Walls manual uses "deg" when describing shapefile outputs.

Somewhere along the way, the project I work with implemented "DEG" as part of the standard header, as in:
#UNITS order=DAV D=Feet A=Deg V=Deg S=Feet AB=Deg VB=Deg
hence four warnings just for that one line, times thousands of such lines.

[Olly Betts]

Let's go for skipping the warning for DEG then, at least as a short term fix. If more cases of reasonable uses crop up we can finesse the rules here further. Fixed in 3d6e8ccfe97168dcd0d2c86f69ff3867ce2fba58.

[Joe Walko]

I bulk-edited working copy of the datasets, changing <<DEG>> to <<DEGREE>> prior to your latest fix, but the results should be the same.

With the various fixes and enhancements you've implemented (and my removing erroneous TAPE=SS), I'm now able to get clean Cavern compile of this large Walls dataset, along with a manageable list of 'warnings'. Survex, imo, is now able to successful process this dataset, superseding my 'documentation comments' to the contrary.

The list of Walls stations vs Survex stations agree except for four or five differences I need to chase down (out of 106,407 stations).

I've not yet done a station by station coordinate comparison. But some rough stats: the maximum absolute difference between Walls and Survex coordinates is 27m for X; Y is 19m, and Z is 9m. For reference, the system is roughly contained in a square which is 11km E to W and also 11km N to S. Elevation span is about 140m.

I'll do more detail analysis when I have some time; but in the short-term I need to focus on some other deliverables which I've been neglecting.

[Olly Betts]

Replying to Joe Walko:

I bulk-edited working copy of the datasets, changing <<DEG>> to <<DEGREE>> prior to your latest fix, but the results should be the same.

To DEGREES presumably?

With the various fixes and enhancements you've implemented (and my removing erroneous TAPE=SS), I'm now able to get clean Cavern compile of this large Walls dataset, along with a manageable list of 'warnings'. Survex, imo, is now able to successful process this dataset, superseding my 'documentation comments' to the contrary.

Great.

The list of Walls stations vs Survex stations agree except for four or five differences I need to chase down (out of 106,407 stations).

That could be due to some corner-case parsing difference, or some difference in handling connectivity (so Walls and Survex have slightly different dangling survey legs). Or something else, but those are two that come to mind.

I'd suggest trying to resolve this first as (a) it's probably easier, (b) whatever is causing it could well affect the coordinates of other stations and (c) a lot of the coordinate difference may have a known cause (see below).

I've not yet done a station by station coordinate comparison. But some rough stats: the maximum absolute difference between Walls and Survex coordinates is 27m for X; Y is 19m, and Z is 9m. For reference, the system is roughly contained in a square which is 11km E to W and also 11km N to S. Elevation span is about 140m.

Much of that could be due to not currently applying grid convergence unless the magnetic declination is calculated from the model. If we assume only the middle of the survey is fixed, then 5500m at 0.3° would be just under 29m. Not sure what grid convergence is like where this cave is, but the magnitude seems broadly plausible.

The coordinates will never be completely identical for a complex cave as it seems Walls sets variances proportional to the leg length, but Survex models the error based on the leg geometries, including covariances.

I'll do more detail analysis when I have some time; but in the short-term I need to focus on some other deliverables which I've been neglecting.

OK, I won't be able to do much for the next few weeks anyway.

[Olly Betts]

Replying to Olly Betts:

Much of that could be due to not currently applying grid convergence unless the magnetic declination is calculated from the model. If we assume only the middle of the survey is fixed, then 5500m at 0.3° would be just under 29m. Not sure what grid convergence is like where this cave is, but the magnitude seems broadly plausible.

I just realised this is easy to correct for in Walls data so I've pushed c36617f08037a1c697fd40f4e98866f617e4a3e1 with that done.

In native Survex data we need to add a way to specify a location to calculate convergence at, but in Walls data the WPJ already specifies one.