Allow suppression of frontsight/backsight 'disagree' warnings for Walls data

[Joe Walko]

Walls compass & clino tolerance settings should be honored. If that's not practical, then a method is needed to enable full suppression of frontsight/backsight 'disagree' warnings for Walls data.

Per documentation: "In TYPEAB= and TYPEVB=, the threshold is ignored...Survex uses a threshold based on the specified instrument SDs, and averages foresights and backsights." Also ignored, evidently, are the Walls default tolerances of 5-degrees for compass and clino.

This can result in massive 'spamming' of warnings triggered by compass or clino fs/bs disagreements, ones are within the Walls default or configured tolerances.

The multi-line format used for the warnings/errors log complicates sorting, or other convenient means to discards these fs/bs disagree warnings in order to find other type of warning which may be important.

A large Walls dataset I work with generates 1,323 compass/clino disagree warnings, 60 compass data on plumb shots, and 11 other types of warnings.

[Olly Betts]

A simple solution for supporting the Walls backsight checking thresholds is to set the compass+backcompass SDs based on the TYPEAB tolerance value (and similarly for clino/TYPEVB).

I've tried it and it works, so I'll probably go for that for now at least. I need to update the docs and add some testcases to the testsuite.

I've not done much surveying anywhere that uses backsights as a matter of course, but my understanding was they were mainly intended to catch mistakes in-cave. If you find readings out of tolerance once you're out of the cave then none of the options seem great. You can plan to go back and recheck the affected readings or resurvey to replace them if the stations aren't refindable, but for expedition caving it could be months or years until you can and it may not even be feasible at all.

Sometimes it may be obvious which reading is wrong from the sketches or other legs, but in general you don't know which to trust, and setting the SD based on the foresight/backsight disagreement at least reflects the confidence you realistically have about the actual bearing of that leg. If such a reading is involved in a loop closure then it will be trusted less than other readings, which seems appropriate.

We could decouple the thresholds from the specified SDs, but that doesn't actually seem better.

[Olly Betts]

Fixed by 35a7d57d63cfbf97d72ea486303359dbc67ad292.

[Joe Walko]

Sounds great - this should help quite a bit with our fs/bs disagreements cluttering up the log; which makes it cumbersome to track down other "Warned" issues.

I know it's mostly a U.S. thing, but most surveys in these parts recorded fs & bs. The trend of late is triple front-sights instead of fs/bs, but the projects I help with mostly still do fs/bs. For the largest project, about 98% of 114,717 shots have bs azimuth data recorded in Walls.

Most common issue this seems to help with is flagging mis-keyed data, which gets corrected right away (hopefully) after Walls warns about the disagreement. Otherwise might never have been caught. For cleanup of historic surveys, it's been helpful when dealing with ambiguous handwriting which originally is entered into Walls incorrectly.

[Olly Betts]

Triple measuring the leg seems the norm with electronic all-in-one devices, and the device can check they're in tolerance in the cave for you. If out-of-tolerance data makes it out of the cave, having three (or more) readings means you can almost always spot the bad one.

60 compass data on plumb shots,

FWIW, I'd suggest replacing the compass reading with -- to silence those (after checking that the data isn't mis-entered of course), perhaps noting the adjustment in a comment for the benefit of anyone later comparing the data with the original notes. Walls is presumably also ignoring the compass reading (just without saying anything) so that should process the same.

and 11 other types of warnings.

Any other false positives?

The one I see quite a lot from some datasets is "Unused fixed point", which is warned about as it could be a mistyped station name in the #fix command, which could go unnoticed if there are any connections from the intended station to other fixed points.

However, it's also reasonable to have fixed points that aren't currently used.

Any other mention counts as a "use", so you can suppress this with #note or #flag, but I don't think there's a way to use either in a way that's guaranteed free from side-effects in Walls.

Maybe a dummy isolated LRUD has no effect in Walls, though it's ugly:

#fix XYZ789 1 2 3
XYZ789 <--,--,--,--,0>

(I see that isn't currently treated as a "use" by cavern, but that's easy to fix.)

[Joe Walko]

I messed something up with my earlier pass through the log produced by cavern with the large datsaset.

Below are updated counts, and more details on "unknown" and "ignoring" warnings, which all appear valid.

Regarding shots with compass data on plumb shot: There are 271, not the ~60 I said earlier. I don't currently screen for those. Most all have bearing of 0, 0/180, or some variation. Leaving about 30 with "real looking" bearing values. Those 30 or so I need to check, as that's a bit of a red flag.

I screen for unused shots and control-points by building the network 'graph' in python - I didn't check every station or control point, but your list looks like it matches mine.

#####################################
"Bulk" warnings:
Warning # Shots
Compass disagree: 1974
Clino disagree: 851
Compass on plumb shot 271
Referred to but never used: 44
Unused control points: 22 (these are mostly unconnected karst features of interest, which might eventually connect)

#####################################
"Individual" warnings: (all appear valid)
Unknown command "Grid"
Unknown command "Grid"
Unknown command "grid"
Unknown command "inch"
Unknown command "INCH"
Unknown command "# "
End of line not blank BY10' <10,20,3,0> 'derived from sketch 1/10 Klausner
End of line not blank #date 11/26/1993-11-26 (Walls parses as 1993-11-26)
Ignoring "-" C5 C6 4.2 141/322 -13.774 <0,2,1.5,3.5>-
Ignoring "." N2 N3 19.05 233.9 5.3 <6.6,2.7,1.8,27.6>.
Ignoring "." FF9 FF10 10.5 354.4/174.5 -7.4/+7.4 *0.5 3 2.5 2*.
Ignoring "+" B1 B2 16.98 200/19 -34/+33 *0 20 100 5*+ #seg /X
Ignoring ">" B1 B2 22.2 17.0/-- 3/-- <1,3,0,1>>
Ignoring ">" N56 N57 35.3 311.5/131.0 3.0/-3.5 <15.5,3,5,3>>
Ignoring ">" D10 D11 72.6 295/115 1/-1 <20,4,13.8,3,.5>>
Ignoring ">" C5 C5' 0.5 -- +90 <1.5,0.5,4,3.5>>
Ignoring ">" W1 W1' 2 39/218.5 -90 <2.5,0,1.5,2.5>>
Ignoring "0.5*" NDL17 NDL18 1.7 87.9/267.9 48.9/-48.9 *2,1,5,*0.5*
Ignoring "3" 1223:Z11 L1 33.5 45/224 +1/0 *15 0 3 1.5*3
Ignoring "5*" M3 J6 7.6 291.5/111.5 -12/11 *2.5 0 7 3.5*5*
Ignoring "6*" P7 P8 16.4 043/224 +5/-3.5 *0 9 4.5*6*
Ignoring "s" H13 H14 7i1 0 90 *0 2.5 9 6*s
#############################################

[Olly Betts]

Thanks, that's a very useful summary.

Most all have bearing of 0, 0/180, or some variation.

Hmm, the Compass reading given on plumbed leg warning already has a check that the compass reading isn't 0 and that the backcompass reading isn't 0 or 180 so those shouldn't be warned about. (They are also commonly seen in Compass .DAT files, and seem far more likely to be a dummy value than a mistake.) Can you provide some examples?

Unknown command "grid" Unknown command "inch"

That's really "unknown to Survex" in case it isn't clear. I was lazy and just reused an existing message, but we probably should instead warn Ignoring unsupported Walls option or similar.

I think inch= could be supported without much work (in the same way that IH/TH are now handled with TAPE=IT). We don't warn about inch=0 (since ignoring that gives the desired result) so I'd expect ignoring these actually affects the output coordinates.

grid= doesn't seem useful with geo-referenced data - are those actually doing something useful, or are they a left-over from a previous era?

The rest do indeed seem like valid warnings.

[Joe Walko]

It does appear that shots with compass data of "0/180" are being excluding from plumbed shot warnings, based on none of those in the log file. But bearing fs of "0" without bs, and 180/0 are warned.

....\CFSB728.srv:16:12: warning: Compass reading given on plumbed shot

M1 M2 24.1 0 -90/90 *5 4 4 8*

-- ....\FSB3987.srv:26:13: warning: Compass reading given on plumbed shot

Z10 Z11 2.1 180/0 -90/90 <8,12,4,0.5>

....\FSB5097.srv:22:18: warning: Compass reading given on plumbed shot

AA19 265:A21 1.5 0 90/-90

---

#UNITS GRID usage is atypical, and lack of support for it is likely not an issue.

#UNITS GRID was used for a couple of theodolite surface surveys where it was necessary to rotate something like 21 degrees. Those surveys are not-normal, and they don't tie to the cave. I use GRID in testing to force abnormal rotation in order highlight objects which are not getting rotated (such as LRUDs with explicity facing azimuth]

---

Looks like INCH was used a couple times used for single shot elevation adjustment - IH/TH mighty have been more appropriate. Or a fake shot. This was not used to deal with systemic elevation change, which is the suggested use case in Walls manual. Other instances were INCH=0.

---

I checked most of our plumbed shots which have "real-looking" compass bearings (I dump the parsed shot data to a spreadsheet which I run screening script, so I can filter & sort, which helps a bit, though for these I still needed to check the survey book scans).

All but a couple are the consequence of two folks who entered each shot containing delta-height data as two separate shots, instead of entering them using Walls' IH/TH method.

First shot representing the horizontal component, using the regular shot data for distance. The 2nd shot is vertical, with the IH/TH data used as the vertical shot distance. They included the original shot-bearing values for both the horizontal and vertical shots. Perhaps a way of "signaling" these were originally single delta-height shots. Who knows :-).

The correct fix is not to remove the bearings from the vertical shot, but rather redo these as single IH/TH shots. Which I may do at some point, but lower priority than other issues.

[Olly Betts]

But bearing fs of "0" without bs, and 180/0 are warned.

I'm surprised by the former - I'll look into that.

The latter isn't included in the current check, but only because I limited it to what we'd seen in real data - it seems reasonable to exclude that as well.

Looks like INCH was used a couple times used for single shot elevation adjustment - IH/TH mighty have been more appropriate. Or a fake shot. This was not used to deal with systemic elevation change, which is the suggested use case in Walls manual. Other instances were INCH=0.

It was indeed easy to add so I've done so.

Ignoring unsupported Walls option

This improved message is now reported for GRID=.

It's now also given for UV=, UVH= and UVV= which we were quietly ignoring but it seems more helpful to alert the user. I haven't seen real world use of them (presumably people prefer the more targetted per-leg variance overrides), but perhaps they feature in your data.

[Joe Walko]

We have 5,099 occurrences of #UNITS UV usage in the main project. Logging them as warnings will not help with my whining about the volume of warning messages :-).

Most occurrences are via use of a 'grade' macro in the SRV header, which is standard practice for this project. Typical SRV will contain something like "#UNITS UV=$(GRADEA)"

Those macros are defined in the top-level project properties for each of the four* datasets comprising the system (.OPTIONS directive in WPJ), and vary slightly:

• $course1=1 $calibrated=2 $nocourse=3.8 UV=3
• $GRADEA=1 $GRADEB=2 $GRADEC=3 $GRADED=5 $GRADEF=10
• $GRADEA=1 $GRADEB=2 $GRADEC=3 $GRADED=3 $GRADEF=10
• $GRADEA=1 $GRADEB=2 $GRADEC=3 $GRADED=4 $GRADEF=10

The grades are based on whether or not the surveys used backsights, used a clinometer, were delta-height surveys and if calibrated compass-course was run before the survey. The validity of this approach is questionable, but that's a separate issue.

Overides with higher UV values are occasionally used. Only a handful of UVH or UVH.

*(I merge the four separate dataset WPJs into a single WPJ via python script, tweaking a few things in the process. Counts and stats I reference here and in other issue threads, unless I note otherwise, are from processing the merged WPJ in cavern)

[Joe Walko]

But bearing fs of "0" without bs, and 180/0 are warned.

I'm surprised by the former - I'll look into that.

For plumb shot with compass fs of 0 and no bs, will this not trigger the warning if compass bs is absent, thus equal to HUGE_REAL?

(backctype == CTYPE_INFERPLUMB && (VAL(BackComp) != 0.0 && fabs(VAL(BackComp) - M_PI) > EPSILON)))

[Olly Betts]

We have 5,099 occurrences of #UNITS UV usage in the main project. Logging them as warnings will not help with my whining about the volume of warning messages :-).

We could easily not warn for UVV=1, etc, which is what $(GRADEA) or $(course1) will expand to - hopefully a lot of surveys are rated with those.

However, I think they're still likely to be more noisy than useful at this point. A reason for adding warnings was to try to find out if they were used much, but you've answered that!

Is it only UV= that has significant use, or are UVH= and/or UVV= used much? Handling UV= could be done by just scaling all the SDs by the square root of the value specified. The only wrinkle is is would mess with how I've implemented the backsight tolerances (so UV=.25 would halve the backsight tolerance being applied). Perhaps it really needs to be applied after the leg covariance matrix has been calculated, or the backsight tolerances achieved a different way.

Replying to Joe Walko:

But bearing fs of "0" without bs, and 180/0 are warned.

I'm surprised by the former - I'll look into that.

For plumb shot with compass fs of 0 and no bs, will this not trigger the warning if compass bs is absent, thus equal to HUGE_REAL?

(backctype == CTYPE_INFERPLUMB && (VAL(BackComp) != 0.0 && fabs(VAL(BackComp) - M_PI) > EPSILON)))

I've not followed the control flow but that seems very plausible. Perhaps backctype should be CTYPE_OMIT for this case, like for an explicitly omitted clino or backclino. I try the examples and see what makes most sense.

[Olly Betts]

Oh I see, the warning suppression fails for the first and third cases because there's a mismatch between which clinos and compasses have values. That's not hard to fix.

I think we should also take the simple approach of not warning when the clino is 0° or 180° and the backclino is 0° or 180°, and not require them to be opposite one another when both are specified - it seems reasonable to use 0/0 for the dummy compass on a plumbed leg, and warning for it and/or 180/180 isn't realistically going to catch any mistakes in the data.

[Joe Walko]

We could easily not warn for UVV=1, etc, which is what $(GRADEA) or $(course1) will expand to - >hopefully a lot of surveys are rated with those.

You meant UV=1, not UVV=1, correct?

But, sigh, I've got over 2,000 surveys with UV other than '1' for this project. We've got lots of old surveys which are assigned grade other than GRADEA (1) due to lack of bs, etc. Also, the default UV gets set to "3" in Compile Options; the concept being surveys not explicitly graded get a lower UV.

There are a dozen or so cases where UV is set to a value >10. A Very Big Number in all those cases, with the intent of pseudo-floating the entire SRV (or section of SRV).

WRT to UVH and UVV - only 8 and 10 occurrences of those, respectively. Rarely used.

It's debatable how much work should go into supporting these sorts of esoteric features which, for all I know, are only used for this project. On the other hand, it is a *really* big project (over 700km).

This UV scheme predates me by decades; it seem over-engineered I've never been able to track down any supporting documentation. When I get a chance sometime, I'll change all the GRADE macros to "1", and the default in CompileOptions, just to see what kind of difference it makes.

---

I'm think a no-warn for those values which are almost certainly dummy/filler data makes sense.

[Joe Walko]

the concept being surveys not explicitly graded get a lower UV.

err...I meant higher UV.

[Olly Betts]

Replying to Joe Walko:

We could easily not warn for UVV=1, etc, which is what $(GRADEA) or $(course1) will expand to - >hopefully a lot of surveys are rated with those.

You meant UV=1, not UVV=1, correct?

I actually meant any of these options - the default for all of them is 1 so doesn't really need warning about (even if it's being used to reset from a different value, that other value will get warned about).

But, sigh, I've got over 2,000 surveys with UV other than '1' for this project. We've got lots of old surveys which are assigned grade other than GRADEA (1) due to lack of bs, etc. Also, the default UV gets set to "3" in Compile Options; the concept being surveys not explicitly graded get a lower UV.

The same option parsing is used for options stored in the project file (which I assume is what "Compile Options" maps onto) so that'll warn too currently, though it's only one extra warning:

wallsdecl.wpj:4:19: warning: Ignoring unsupported Walls option “UV”
 .OPTIONS $macro=1 UV=3
                   ^~

It's debatable how much work should go into supporting these sorts of esoteric features which, for all I know, are only used for this project. On the other hand, it is a *really* big project (over 700km).

I'll have a look as it might be easy to implement now. If not I'll turn off this warning for UV= for the time being (which just puts things back as they are in the current release). 2000 warnings is an improvement on 5000 but still far too many.

[Joe Walko]

The same option parsing is used for options stored in the project file (which I assume is what >>"Compile Options" maps onto)...

Correct; I was referring the UI "Compile Options", accessible in "Properties" after right-clicking in object in the tree, which are verbatim from the WPJ ".OPTIONS" pseudo-directive, which mostly is a synonym for #UNITS directive.

So-called "Compile Options" are among my Top 5 Walls complaints. Typical Walls users are oblivious to them, 'out of sight, out of mind', and are unaware of them in cases where they inherit projects where someone used 'Compile Options.

Hence the over-the-top spreadsheet I generate when 'screening' the datasets which makes a point to present all possible sources for, example, declination adjustments, and how the final value was determined (see attached screen-snippet JPG).

[Olly Betts]

Replying to Joe Walko:

The same option parsing is used for options stored in the project file (which I assume is what >>"Compile Options" maps onto)...

Correct; I was referring the UI "Compile Options", accessible in "Properties" after right-clicking in object in the tree, which are verbatim from the WPJ ".OPTIONS" pseudo-directive, which mostly is a synonym for #UNITS directive.

So-called "Compile Options" are among my Top 5 Walls complaints. Typical Walls users are oblivious to them, 'out of sight, out of mind', and are unaware of them in cases where they inherit projects where someone used 'Compile Options.

I found it interesting that very early versions of Survex had .svc files which were conceptually somewhat like .WPJ files, but we concluded it was better to scrap them and allow use of a top-level .svx which included other .svx files instead, which means users only need to learn one file format for raw data. (A change like that would be harder for Walls though as the .WPJ includes information like which branches of the survey tree are expanded in the UI; I think the .svc had settings and .svx files to include. Also .WPJ couldn't just be scrapped at this point of course - Survex was new enough at that point that it wasn't widely used.)

It seems particularly unhelpfully that the .WPJ format is textual but pretty opaque - you can open it in a text editor but a lot of key information is encoded in .STATUS codes which are decimal numbers of (mostly) bit flags (which David doesn't seem to have documented the meaning), and the nesting isn't reflected by indentation. Also .NAME has a different meaning depending if it's in the context of a .BOOK or a .SURVEY.

Compass has .MAK files which are perhaps more like .svc were - they're intended for humans to read and write and IIRC they don't encode UI state, but the format is somewhat cryptic (and unknown commands are quietly ignored by Compass). I think they must suffer from a similar "out of sight, out of mind" problem as non-US Compass datasets pretty commonly use NAD27 (I've had to add special handling to Survex to handle these as there are only EPSG codes in the defined area of use).

Hence the over-the-top spreadsheet I generate when 'screening' the datasets which makes a point to present all possible sources for, example, declination adjustments, and how the final value was determined (see attached screen-snippet JPG).

So is DECL unused and the declination instead manually applied via INCA and INCAB? If so, why is it done that way?

[Joe Walko]

So is DECL unused and the declination instead manually applied via INCA and INCAB? If so, why is it done that way?

tl;dr: INCA/INCAB include both instrument correction and magnetic declination adjustment based on star-sighted 'compass course' pre-survey instrument readings

Prior to survey trips for this project, folks are expected to "run the compass course" with instruments which will be used. The "compass course", aka CC, is star-sighted at a bearing of 305.8°, and is level. FS and BS compass-course reading are taken, and averaged after adding 180° to the BS readings. The difference between the instrument's CC averaged FS & BS readings and the reference CC reference value of 305.8° are then used for the INCA and INCAB values. Likewise with INCV and INCVB.

Compass-course FS vs BS is expected to be within 1° for "legacy" instruments, and 0.5° for "electronic" instruments. Not always enforced; sometimes alternate instruments are not available.

A recent survey recorded pre-survey CC compass FS and BS readings of FS=310.65 and BS=131.075. The 131.075 BS +180=311.075, a 0.425° FS/BS delta.

(310.65+311.075)/2=310.8625, and 305.8-310.8625-305.8=-5.0625, thus both INCA and INCAB get set to -5.0625. Simiiliar process for clino.

DECL gets set to 0°, and #DATE must preceed #UNITS to avoid #DATE-based magnetic declination supersceding DECL=0, effectively resulting in doubled magnetic declination correction.

Thus, for the above example, the SRV header contains:
#Date 2026-03-12
#units decl=0 Inca=-5.0625 incab=-5.0625 (along with incv and incvb)

These inca/incab values reflect both instrument correction factor *and* magnetic declination adjustment for this area. Which was not McKenzie's intent for INCx, which ostensibly should contain only instrument correction factor; magnetic declination should be in DECL. It'd be possibly to modify the compass-course process to isolate instrument error from magnentic declination. But that'd make things even more complicated, and would likely introduce errors. Folks stuggle with the current process, and are known to fudge things. Note that we don't advertise the compass course reference value, as that info would bias some CC data; newbies don't understand the CC data encompasses magnetic declination.

If no compass-course data is available, then #DATE occurs *after* #UNITS, and Walls' IGRF magnetic declination is used. For example:
#UNITS DECL=0 INCA=0 INCAB=0
#DATE 2026-03-12

Some surveys in the 1970's and 1980's used a static table for "compass correction factors" for specific compasses commonly used for the project. In those cases, INCA/INCAB are set to the predetermined compass correction factor, and #DATE is also used to set IGRF magnetic declination adjustment. For example:
#UNITS DECL=0 INCA=+1.5 INCAB=+1.5
#DATE 1976-08-08

Hence, three possible permutations of #UNITS DECL INCA INCAB and #DATE.

[Olly Betts]

Interesting - CUCC has/had calibration points for the Austria expeditions (a series of them as the active surface camp moved around, but the same ones reused for many calibrations). I'm not sure if they're still used or if they now rely on electronic instruments having been calibrated as a separate step. Back before we had IGRF support we looked at using the average of the calibration measurements from this directly, but Wookey concluded it would actually introduce more error than it eliminated and instead used the data en masse to identify e.g. instruments that needed a calibration offset, people who were routinely reading instruments badly, etc. You probably suffer less from having quite a turnover of expedition members, most of who are new to surveying, which probably makes a significant difference.

Survex should already handle #DATE and DECL= overriding each other correctly.

Note that grid convergence doesn't currently get applied when the declination comes from DECL= though (Eric already opened #140 for that).

[Joe Walko]

Survex should already handle #DATE and DECL= overriding each other correctly.

I remember reporting that issue through Eric early on when I helped test a while back, and it was fixed soon after - it's one of the first things I always check with Walls parsing, since it's, um, goofy. #DATE fundamentally is a synonym for #UNITS DECL=<IGRF-Declination> if date-based declination is enabled.