Do not silently ignore Walls shot data containing Instrument Height (IH) and Target Height (TH)

[Joe Walko]

Walls IH/TH data should be processed, not silently ignored. This is a defect, not just an enhancement, as Walls shot data is ignored with no warning to the user.

Many older compass and tape surveys were performed without a clinometer, instead the vertical offset of the "to station", relative to the "from station" was measured via tape, and records. Sometimes referred to as "flat tape surveys". A large Walls dataset I work with contains over 3,000 shots with IH/TH data.

See attached bundle of sample test data and related artifacts.

[Olly Betts]

Yeah, we really should at least complain here. The main reason we currently ignore this is because it looked rather fiddly to check and there were a lot of other Walls things to support which were actually used in the datasets people had provided me with. We can't just complain if we see tape=<something other than ss> because Walls defaults to tape=it. So we need to actually parse the heights and complain based on them and the current tape setting.

Since then I've added support for Walls diving surveys and I think that probably provides most of the needed machinery to properly check this. Thanks for the sample data - that'll help a lot.

Many older compass and tape surveys were performed without a clinometer, instead the vertical offset of the "to station", relative to the "from station" was measured via tape, and records. Sometimes referred to as "flat tape surveys". A large Walls dataset I work with contains over 3,000 shots with IH/TH data.

Is this the only way the taping stuff is used in practice (other than for diving data)? I'm wondering if the "Hybrid c&t shot" is a hypothetical example or something people actually use.

We might be able to simply map "flat tape surveys" to Survex's "cylpolar" style (similar to how we map TAPE=SS ORDER=AD and TAPE=SS ORDER=DA to the Survex's "diving" style).

[Olly Betts]

Just to check I have understood how these modes are meant to work, is this correct?

• If there's a clino reading, Survex currently mishandles unless instrument and target heights are the same

And depending on the tape setting:

• tape=SS: No other considerations
• tape=IT: Survex currently mishandles unless instrument and target heights are the same
• tape=IS: Survex currently mishandles if instrument height is non-zero
• tape=ST: Survex currently mishandles if target height is non-zero

If so, specifying instrument and/or target height when tape=SS and there's no clino should probably warn as those heights have no effect. Similarly for tape=IS, no clino and a target height; and tape=ST, no clino and an instrument height.

The "flat taping" use case I think I understand now (though how do you know you're actually taping level? A small angle off level changes the tape length very little, but any error in levelness is going to appear in the measured instrument vs target heights isn't it?)

Are IS and ST modes actually used? If so, do you have examples of usage?

[Joe Walko]

I'll try to provide responses to questions, and provide more info when I catch up with some backlog. In them meantime, to provide some context, attached is sample, one including hybrid angular inclination combined with IH/TH, from 1981, of a standard 'survey form' used by a cave research group in KY. The 'form' was a rubber ink stamped overlaid onto a 6"x4" spring-bound "memo book", a common format in the U.S.

Note the form has columns for "Vert Angle (°)" and "Vert Dist (FT)". Survey trips typically recorded either inclination in degrees OR vert distance, reflecting the height of the "to station" relative to the "from station". Previously I mentioned surveys which record only "vert dist" as "flat tape" surveys - a more common term, one I use, is "delta height surveys.

This particular survey book was from a survey training exercise. Four separate books from the class, with identical data, confirm the "hybrid" shots it contains were intentional, not recording blunders.

Maintaining original "stations" is a strong goal for resurveys in this cave system. In this example, it looks like recent collapse/breakdown obscures what previously were line-of-sight shots for X3->X4 and X4->X5. If I'm interpreting this correctly, the workaround, one which presumably was taught to be standard practice during this training, was to shoot to an alternate point 1/2 foot above X4, recording both the inclination and the 1/2 ft vertical offset, followed by inclination and 1/2 down from X4 to X5. P5->X1 involved similar shenanigans - note in sketch states "station is 1.5ft off the floor".

The hybrid shots in this survey thus appear a bit contrived - Why didn't they just add in new intermediate stations? Nevertheless, there is a scattering of such hybrid shots, ones not from training, in this large dataset. They are rare, but exist. While mostly confined to old surveys, one well known cartographer continues to occasionally use these hybrid shots (much to my chagrin - this trip an alert when I run my 'screening' script).

(sample survey book image attached)

[Joe Walko]

samples images from survey book with hybrid shots and "vert dist" columns

[Joe Walko]

I've got three tape=SS surveys which are underwater 'dive' surveys. I'll get you a sample of one of them.

Along with a couple hundred tape=ss surveys, from the same project, which are regular dry land surveys ("dry" being a figure of speech here). It would appear, for a period of time, tape=ss s was the 'standard' Walls header for one of the four ridge datasets involved. Or, a victim of endless propagation of "copy/paste the last header we used" after one on the dive surveys. I'll need to look into this.

I tested a couple, changing them to tape=it, and there was no change in XYZ station coordinates. The one's I tested with each had a few IH/TH shots. I'll need to experiment a bit.

[Olly Betts]

I've just pushed 4367e8d1cba6229e8bd477b899d043beadae5d44 which maps TAPE=IT with either ORDER=DA or ORDER=AD to Survex "cylpolar" data style.

This case is exercised in one survey in the Thailand dataset I have. I compared the output with Walls on the same file and the plan matches and the station elevations are the same (to the 1 d.p. precision which Walls GUI shows, but that's enough to demonstrate this is a correct mapping, and that I don't have a mistake like a reversed sign on interpreting the heights).

This is clearly only a partial solution, but it seems useful to do anyway (e.g. it means these legs get flagged as being a different data style in the .3d file).

[Olly Betts]

It looks like the examples I have that aren't handled by current Survex git main are all TAPE=SS ORDER=DAV and all have the instrument height as --.

If anyone actually uses TAPE=IS or TAPE=ST then having some real (or realistic) examples of those would be particularly useful. If they were just added for completeness and seem to be unused in practice then we can just warn if we see them so if someone has made use of the (deliberately or accidentally) it won't be quietly processed in a different way to Walls.

[Joe Walko]

I have over 200 TAPE=SS surveys in the dataset I work with, which has over 5,000 surveys. All but 20 of these TAPE=SS surveys have no IH/TH data, thus TAPE=SS vs TAPE=IT has no impact.

Of the 20, a couple are dive surveys; the rest are on land. The ones on land typically have "--" for IH. I’m researching this, trying to figure out why TAPE=SS is being used. Is it intentional? Is it result of copy/paste laziness? This will take a while for me to sort out.

The rest use Walls’ default TAPE=IT; there are no TAPE=IS or TAPE=ST surveys in this dataset.

---

For delta-height surveys, where IH=0, it is a common shortcut to enter inverted TH data into IH and omit the TH column.

For TAPE=IT surveys, Walls manual notes that vertical rise = (IH-TH). Thus, a delta-height survey where the TO station is five feet higher should contain (IH=0 or --) and (TH= -5), since 0-5=+5. Note the reversed sign: plus5 elevation change gets entered as negative5 for TH in Walls. This is the technically correct method. And also incredibly unintuitive and confusing.

Folks entering these surveys figured long ago they can get the same result by entering +5 for IH, and nothing for TH. Since +5-0 = +5. Technically incorrect. But the majority of delta-height surveys in this dataset are using this technically incorrect method.

TAPE=SS also calculates vertical rise via (IH-TH), afaik.

---

I’ve noticed that the distance specified in shot data for TAPE=SS is what's used as the actual length of the shot, in Walls. Whereas, with TAPE=IT, the Walls length combines the shot distance + the vertical offset length.

For flat-tape survey shots, which are effectively right-triangles, with TAPE=SS the Walls shot length = the shot “distance” data. The length of the hypotenuse will equal shot distance data.

With TAPE=IT, the shot distance data comprises the horizontal leg, and the net Walls shot length = sqrt(d^{2 + (ih-th)}2) - Pythagorean theorem. (I have diagrams which are clearer than what I can write - can't find them atm.)

---

Going back to my TAPE=SS dry-land delta-height surveys: Are folks shooting the diagonal (hypotenuse) distance instead of the horizontal distance? If so, then TAPE=SS works. But seems unlikely, as these are surveys using clino/degrees for most shots.

---

There is near 100% chance I've gotten something wrong above :-)

[Olly Betts]

I've now implemented support for instrument/target heights with TAPE=IT in 2de1873c330e8c965d847018ca51d6ad54a78f16. Your example now processes to give all a-, b- and c- stations the same coordinates.

I’ve noticed that the distance specified in shot data for TAPE=SS is what's used as the actual length of the shot, in Walls. Whereas, with TAPE=IT, the Walls length combines the shot distance + the vertical offset length.

That matches what I understood from the Walls docs.

Going back to my TAPE=SS dry-land delta-height surveys: Are folks shooting the diagonal (hypotenuse) distance instead of the horizontal distance? If so, then TAPE=SS works. But seems unlikely, as these are surveys using clino/degrees for most shots.

It seems most likely TAPE=SS was just "cargo culted" based on seeing it used in another survey (maybe happening over and over, originating from a diving survey).

Without knowing the actual motivation for the various TAPE= methods, I'd guessed that TAPE=SS might be used when using a tripod or stick or similar to allow more comfortable instrument sighting from vertically above the station, with the tape being measured from the station itself (perhaps because the tape was measured by different people to the instruments). I don't know if anyone actually surveys like that though.

[Olly Betts]

93d3e30f75622365a2e4aa3b472f4f7516ce5b31 adds a warning for unsupported TAPE= + ORDER= combinations when a non zero height is specified, for examples:

./badopts.srv:26:18: warning: Instrument and target heights currently ignored with Walls option combination TAPE=IS ORDER=DAV
 1 6b 3.00 004 05 -- 1.0
                  ^~~~~~

One of the datasets I have has some TAPE=SS ORDER=DAV legs, so I can take a look at those too.

(I think I can suppress this warning for TAPE=SS whenever the heights are equal, but I'll see if I can just support any TAPE=SS before worrying about that.)

I think the combined changes so far address the original report since we should now either process a leg with instrument and target heights like Walls, or warn that we're ignoring them.

Beyond that I can try to support any remaining combinations which are actually used, but it'd be helpful to have an example of each case that's useful (can be real or artificial - your 3:4:5 triangles were great).

[Olly Betts]

As per #164, TAPE=SS ORDER=DAV is now handled when there's IH/TH but no clino readings.

I'm going to leave TAPE=IS and TAPE=ST until there's evidence of actual real-world use (we warn if they're in effect and we see a leg with non-zero IH or TH).

The case of TAPE=SS ORDER=DAV with both IH/TH and clinos is still not supported, and seems to occur in practice. From what you've said, maybe TAPE=SS is actually incorrectly recorded for much of that data though. We could probably fudge the tape and clino readings to include the effects of IH/TH, though the trigonometry seems a bit hairy and the error modelling wouldn't be quite right. Maybe warning and then handling TAPE=SS as TAPE=IT for these cases would be a better interim solution than ignoring the heights? That at least seems helpful for cases where TAPE=SS is recorded when it should really be TAPE=IT, and probably gives a more similar leg vector in general (especially when IH and TH are quite different).

[Joe Walko]

• I've not yet have a chance to confirm TAPE=IT surveys with IH/TH data, with or without clino dasta, are resolved; will work on that next week.

• Are mentioned, I've got over 200 TAPE=SS surveys in the large dataset. All but 20 or so have not IH/TH data. I've confirmed the vast majority of these TAPE=SS are a mistake - all but a handful need to be changed to TAPE=IT.

• I still need to go through and understand the remaining valid TAPE=SS surveys; one has no IH data, and some have hybrid shots. Will submit more info once I've done that.

[Olly Betts]

I tried a very simple example of TAPE=SS with a clino in Walls and it gives an error which makes me think either I don't understand how this is meant to work, or Walls is mishandling it.

Consider:

#FIX S1 0 0 0
#UNITS TAPE=SS METERS ORDER=DAV
S1      S2      13.00   000     36.86989764584402       1.00    15.00

The imaginary situation I've measured here is:

S1 to S2 forms a 5:12:13 right-angle triangle (horizontal distance 12m, S2 5m lower).

The instrument and target form a 9:12:15 right-angle triangle (horizontal distance 12m, target 9m higher). The clino reading is the smallest angle in this triangle.

The instrument height is 1m, so the target height is 5+1+9=15m.

Walls complains:

Ambiguity due to |IH-TH| > taped dist! Dist assumed 14.00m (not 13.00m) to match |IH-TH|.

If we're going to adjust the tape and treat this as TAPE=IT then the tape should be 15.00m for my imaginary situation.

I can't find an alternative solution which satisfies all the constraints we have (though I may be missing one - I have a degree in maths but that was decades ago).

If I set IT to 14.00m and work through the right angle triangles then TH is 6.6+1+8.4=16 (assuming S2 lower) or -6.6+1+8.4=2.8 (assuming S2 higher) but it's meant to be 15, so even if there are other solutions, Walls seems to have gone with an incorrect solution.

Am I misunderstanding how TAPE=SS is meant to work?

[Olly Betts]

Looking at Walls .LST report:

 S1 0.000 0.000 0.000
 S2 0.000 13.4400 -3.9200

So it has S1 and S2 14m apart.

However I told it I was measuring the tape station-to-station and that the tape was 13m; there are no loops so shouldn't S1 and S2 be 13m apart?

---

I also tested a similar situation but with a negative clino, and one with no clino (so diving-style) and these both give stations 13m apart in the .LST, which matches my expectations:

S1 S2 13.00 000 -36.86989764584402 5.00 1.00 ; IH and TH both increased 4m to avoid -ve TH
S1 D2 13.00 000 --                 1.00 6.00

It's looking to me like Walls has a check here which seems to only be valid when there's no clino (when the tape should be >= |IH-TH|).

[Joe Walko]

I can't find an alternative solution which satisfies all the constraints we have (though I may be missing one - I have a degree in maths but that was decades ago).

I, on the other hand, could not deal with calculus and other university math. Also decades ago. (Root cause wasn't lack of aptitude, rather a personal incompatibility with structured/linear learning methodologies. Aka "school").

I'm also flummoxed by Walls' handling of TAPE=SS shots with clino data. I *thought* I understood it; I do not, nor do I after fighting with various test cases for quite some time. Cheating, and looking at the source code, did not help much.

Walls is a bit non-deterministic here. Not literally, but the comment "3 cases worth distinguishing" is relevant.

Meanwhile, I've been reviewing the surveys we have with TAPE=SS, particularly those with hybrid shots with both clino and iH/TH. None with clino data are valid. Meaning, none of them should be TAPE=SS. These will get changed to remove TAPE=SS from units. It will take some time to push these changes through the four data managers, but I've already bulk-edited my local version of the data.

I have only two, maybe three legit TAPE=SS surveys. They all are dive surveys, and contain no clino data, only IH/TH. All shots contain both IH and TH - no cases of TH data only, which was an issue mentioned above.

I've no data or survey books that I can reference which contain clino+IH/TH data from "real" surveys. Thus not really able to perform any testing with TAPE=SS & clino & IH/TH using real data, data which would have auxiliary info (trip reports, etc) giving insight to methodology.

[Olly Betts]

I've filed a bug report for a simpler example where Walls seems to incorrectly adjust the tape: ​https://github.com/wallscavesurvey/walls/issues/65 (and linked here).

I've also noted in the .svx format docs that the "cylpolar" style is known as "flat tape" or "delta height".

I don't think there's anything left to address here. We process these in cases which are actually used and warn for cases we don't handle.

We could probably handle the remaining cases if they're actually useful, but it's not trivial enough to do that I want to spend the time to just implement it regardless.

So closing, but if you find anything I've missed please say and we can reopen.

[Joe Walko]

I've bulk-edited my temporal version of the datasets to remove the erroneous TAPE=SS. That leaves me with only three legit TAPE=SS surveys, sump/dive surveys, which do not have clino data.

Cavern compile no longer warns about dodgy TAPE=SS surveys, since I scrubbed those, and IH/TH data seems to now be handled for the "regular" TAPE=IT surveys (though I've not yet done a detailed station coordinate comparison).