Comments on Survex manual, "Walls" , Chapter 8

[Joe Walko]

Some changes should be considered for the chapter on Walls (chapter 8).

[Olly Betts]

Having the comments in a PDF is really unhelpful - please don't do that.

I've cut and pasted the text into the description. Sadly the resulting formatting is poor, but at least now the text is actually in the ticket.

[Olly Betts]

Chapter 8, third page:

The page numbers aren't useful references as the pagination will depend on the format you're looking at (and I'm looking at the .rst documentation sources which don't have any pagination). Luckily the quoted text is enough context.

Pulling out some comments which I think have been addressed in the meantime:

---

“UV=, UVH= and UVV= are all quietly skipped”: A significant limitation which should be noted at start of chapter. It’s not unusual to set UV/UVH to very high values for error-ridden surveys remain in the dataset as they are needed to connect the network. Ignoring the UV/UVH values will change the geometry of the processed cave data

These are now handled. There are some difference to Walls currently (zero is mapped to a small value and large values are clamped) but the results of processing should match better now.

(Am curious why these are ignored, but not variance parm when included on line of shot-data, since they do the same thing).

Mostly it's because I had datasets which used variance overrides but none with UV=, etc. It's much easier when I have a real dataset to test with, and it's also an indication that a feature is actually used.

The variance overrides are also a little easier to deal with because they're on the leg, whereas we need to remember the current UV=, etc settings.

---

In “TYPEAB= and TYPEVB=” paragraph its states “Survex uses a threshold based on the specified instrument SDs”.

I've already rewritten this section (when I implemented TYPEAB=/TYPEVB=).

Nowhere is mention made of how one might define instrument SD in Walls data which cavern is processing. Is it possible, for example, to wrap the Walls data with Survex formatted data in order to set such parameters?

Yes, you could wrap Walls data like that in 1.4.22 (using the usual *SD command to set the SDs, which would then be inherited when processing Walls data).

However now the SDs are set when we start processing a .srv so the backsight warnings match Walls' thresholds (and TYPEAB=/TYPEVB= adjust those SDs for you).

---

“Walls explicitly documents that Unprefixed names”: Observation/opinion, not an issue with documentation: Some unusual characters in station names possibly ought to generate ‘warnings’ in cavern, because it’s such a bad practice. But I need to think about this, and possibly submit an enhancement request.

I try to avoid creating a situation where users have to put up with warnings. It's good to be able to keep a dataset free from warnings, as then it's hard to miss a new warning. If it generates N already, it's easy to miss if that becomes N+1. You'll probably notice eventually, but it's typically easier to address when you've just made the change which caused it because everything is fresh in your mind at that point. So I'd strongly prefer there to be a clear fix for each warning which is not difficult to implement, or a clean way to suppress the warning if it is sometimes wrong.

It's arguably bad practice to use e.g. ( in a Walls station name, but it's documented as allowed and if you have existing data which uses it it may not be easy to change it to use a different character - you have to pick a different one which doesn't clash, and then find all occurrences, and there may be external references to station names in places outside the Walls data.

A warning for # in a station name could be reasonable as it's documented as not allowed but the implementation allows it. This would still suffer from the same problems of potentially being hard to change existing data to eliminate uses.

I don't really see a good way to suppress such naming warnings, short of adding some sort of Survex-specific annotations.

I think the suggestion elsewhere to add a "strict" naming setting to Walls itself would address this in a better way. Once that is added I can easily support it.

---

Survex successfully processed Big Bat Cave, as noted (this statement, in the doc, was based on my feedback to Eric). The same is not true for the Mammoth Cave data; that data contains over 3,000 shots which use IH and/or TH data which Survex ignores. It is not possible to “successfully” process the Mammoth Cave data while ignoring the IH/TH shot data. Lack of errors by cavern, and a result which resembles, at a high-level, what Walls produces does not count as “successful” here.

I'd understood from Eric that we were already doing a decent job on Mammoth too, but it seems we miscommunicated.

I think this will be true for 1.4.23 though?

IH/TH is now handled except for:

• TAPE=IS/TAPE=ST which you've said aren't used in your data, and
• TAPE=SS when a leg has clino and IH/TH which you suspected might be incorrect usage in the data due to copying options from an earlier diving survey.

[Olly Betts]

#SEGMENT – FYI: The “common practice” of using Compass-style ‘shot-flags’ for segment names is due to the large number of Compass files which were imported into Walls. Walls’ converts those Compass shot-flags into corresponding #SEGMENT names. More often than not, the Walls users is not actively “practicing” such usage; often they are just artifacts from the Compass import. Survex #SEGMENT name handling, as documented, is appropriate.

Thanks for the explanation - I've adjusted the text here to reflect it.

---

Assumption regarding Walls’ defaulting to 19xx for two-digit year is correct. See Walls32/wall-srv.cpp below.

I'm avoiding looking at Walls source until the missing licence is resolved, so please don't copy and paste from it.

I failed to spot an easy way to check the results of date handling before, but I just tried processing a one-leg survey with #date 09-09-00, exporting as a vector shapefile and looking in QGIS at the date attribute, and that shows it does indeed get handled as 1900, so I've dropped this from the docs and adjust a code comment about it.

(Incidentally, Walls Compass’ importer handles this differently, year 00-50 results in 20xx while 51-99 results in 19xx. Presumably mirroring Comapss behavior)

The Survex source code has a comment "Note: Larry says a 2 digit year is always 19XX" about dates in Compass .dat files (the format document for it only says "The year can be a two digit or four digit number, i.e. 1992 or 92", but I emailed Larry to clarify various details like this when I was adding support for Compass formats and he was really helpful, so it'll have come from that email correspondence).

I suspect there's not much raw Survey data from before 1951 surviving, so it may not matter. That seems the case in the UK, though maybe US cavers of that era were better archivists.

---

“If a station is used with an explicit Walls prefix...”. Observation/opinion, not an issue with documentation. Regarding statement: “This is currently applied even if the explicit prefix is empty”. Note the so-called ‘empty’ prefixes are often used for stations with project-wide importance. Use of an explicit empty prefix is usually done for a reason.

The main benefit is that aven has an option to show blue blobs on stations flagged EXPORTED (and they will get their name shown in preference when station names overlap and we pick which to show), so stations with project-wide importance very much make sense to flag as EXPORTED - I'll adjust the text to reflect that.

I suspect it is also currently relevant if you want to process a dataset with both .srv and .svx data and connect between them as connecting is only allowed to an EXPORTED station in a pure .svx dataset, unless you use *infer exports (or never use *export).

We turn on *infer exports for Compass data, which means stations there get exported if they're used to connect between two Compass surveys - I think we don't turn this on for Walls data as it would probably interfere with exporting based on explicit prefixing, but I hadn't considered the effect on being able to connect from Survex data.

I should test out this scenario, and probably advise people who want to connect to a Walls station from Survex data to mention that station including its explicit prefix in the Walls data (add e.g. #note ABC:Z12 Survex export if there isn't already a mention). The most likely scenario is probably wanting to combine data in different formats which is in the same area but each has its own fixed points and there are no connections, but there really should be a way to connect if you need to.

[Olly Betts]

I noticed we document Survex currently quietly ignores the optional X argument to TYPEAB= and TYPEVB= which means to check backsights but only use foresights (and that's what the code does). There are no instances of this being used in any datasets I have - do you have any uses?

I think it would be pretty easy to change it to use the foresight and ignore the backsight. The exact Walls semantics might be more of a pain as there's no existing mechanism to achieve it.

[Joe Walko]

I could find no instances in the large dataset of "X" being used with TYPExB to specify use only fs. I was somewhat surprised, as I expected to find some. There may be cases in some secondary datasets; I'll check when I have time, but at this point I don't expect to find any.

The use-cases I expected to find were where separate instruments are used for fs and bs and they routinely disagree (which ought to be addressed in-cave), and it's determined the bs instrument is untrustworthy. Thus, one disregards the compass bs or clino bs data. (We discourage the practice of separate fs/bs instrument, hence such surveys are rare to start with).

But - my imaginary use-case implies there ought to also be an option to use bs only data, and disregard fs data. Which there's not. Thus, I'm a bit vague wrt circumstances where the X option would be used.

Perhaps the more likely use-case are surveys where compass bs data is not consistently 'normal' or 'corrected', but a random mixture of both. It'd be tedious to add a bunch of TYPEAB=C|N directives; easier in the case, to just ignore the inconsistent compass bs data.

[Olly Betts]

Replying to Joe Walko:

I could find no instances in the large dataset of "X" being used with TYPExB to specify use only fs. I was somewhat surprised, as I expected to find some. There may be cases in some secondary datasets; I'll check when I have time, but at this point I don't expect to find any.

Thanks for checking.

The use-cases I expected to find were where separate instruments are used for fs and bs and they routinely disagree (which ought to be addressed in-cave), and it's determined the bs instrument is untrustworthy. Thus, one disregards the compass bs or clino bs data. (We discourage the practice of separate fs/bs instrument, hence such surveys are rare to start with).

If they routinely disagree then it's unhelpful to still report the disagreement when X is specified, so that seems an odd design choice if this is the intended use.

But - my imaginary use-case implies there ought to also be an option to use bs only data, and disregard fs data. Which there's not. Thus, I'm a bit vague wrt circumstances where the X option would be used.

You could swap FS and BS data around if the FS is the dodgy one, but I think the order is always FS/BS so you'd need to actually swap the values in the .srv file which is potentially error prone.

Perhaps the more likely use-case are surveys where compass bs data is not consistently 'normal' or 'corrected', but a random mixture of both. It'd be tedious to add a bunch of TYPEAB=C|N directives; easier in the case, to just ignore the inconsistent compass bs data.

Though again it seems unhelpful to warn about mismatches.

Perhaps we're missing a use-case where you'd want to ignore and still warn, but maybe the seemingly odd design choice here was to provide flexibility for unanticipated uses rather than because they had such a use case in mind. You can presumably use something like TYPEAB=N,360,X to ignore the backsight and never warn.

I'm thinking it's probably useful to warn that we're ignoring X if we see it, as then the user knows and they can tell us they're using it. It's probably not going to drown anyone in warnings.

Update: I have added a warning:

w_fsbs.srv:94:19: warning: Ignoring "X"
 #units TYPEAB=N,1,X TYPEVB=C,1,X
                   ^
w_fsbs.srv:94:32: warning: Ignoring "X"
 #units TYPEAB=N,1,X TYPEVB=C,1,X
                                ^

[Olly Betts]

“LRUD data is currently ignored.” This is a significant limitation and as mentioned above, should be noted earlier in this

BTW, we're actually not far from being able to hook this up. I've got LRUD working for Compass .DAT, and we're now parsing Walls LRUD= options and filling in the settings we need based on them. It's unfortunate that Walls parsing of LRUD seems particularly quirky, and that's the part still missing. At least errors from mishandling LRUD corner-cases are not cumulative.

[Olly Betts]

I'm looking at doing a major restructuring this chapter - as you point out it's currently an unhelpful mix of limitations that a user would definitely want to be aware of (which will be much shorter for 1.4.23) and more obscure details, and the currently order is largely arbitrary.

It's grown to this state piecemeal, and originally I'd kind of assumed that a lot of this content would be temporary as limitations got resolved, but instead it's mostly just got longer and more unwieldy.

I think I'm going to stop trying to document how support has evolved through Survex versions here as that just complicates this chapter making it harder to follow for the normal case where you want to know how Walls support is in the current release. With the next release the simplest and best advice to users wanting to process Walls data will be to use 1.4.23 or later, and so we should state that and just document the status of 1.4.23. The NEWS file describes when support was added for features, and when various bugs or limitations were fixed, so users who really want to run an older version for some reason, or who have been using older versions, will still have access to that information.

[Olly Betts]

Survex limitations which have meaningful impact on how the Walls shot data is processed should be more clearly noted and listed near the start of the document. Existing Walls users considering Survex need to easily gauge if their Walls data is suitable for Survex. How Survex handles arcane, but less impactful, syntax ambiguities should be demoted till later in the document.

I've pushed a major restructuring of the chapter. Let me know if you can see further improvements.

Survex ignores IH/TH data, for example, which impacts many older “flat-tape” surveys in Walls.

This particular one is no longer the case, but your wider point is valid - I've split things into the things users ought to know, the things that probably don't matter to them, and a third section of things Survex warns about which Walls doesn't (where I've attempted to suggest what to do about such warnings).

Intro text: Walls is no longer being developed – No longer true. There is active Walls development, and occasional new releases, via Andy Edward’s github repo (​https://github.com/wallscavesurvey/walls/releases) “...focus of support for Walls formats is primarily to help people with existing Walls data to migrate.”

Oh yes, that was true when written, but is out of date now.

– Opinion: Compiling Walls data to .3d format isn’t particularly helpful for Walls migration as one still needs to use Walls to modify the original data.

The .srv format is human readable and writable text, so you don't need Walls to modify that data.

The .wpj format is text, but not amenable to human edits - key information is encoded as binary bits into decimal numbers, and the object boundaries and nesting are not obvious. It also doesn't help readability that .NAME is used for both the name of a book and the filename of a .SRV file.

However you could migrate to using Survex by replacing the .wpj with a simple .svx file which includes your .srv files using e.g. *include file1.srv. (If you're setting options in the .wpj you'd need to copy those to #units at the start of each .srv.)

Then going forwards, you can either add new data as .srv or .svx.

So I think this really is a plausible migration path (though presumably Walls users are less likely to want to migrate now). It's likely to give a more faithful result than trying to wholesale translate a large Walls dataset to another format.

The real benefit, and more common use-case, is use of Survex as a complement to Walls. Survex offers graphical features not available in Walls, and facilitates use of downstream apps which can consume .3d format.

Sure, but only because Walls is now being maintained again.

Paragraph starting with “Since Survex 1.4.21, the TAPE= option”: I can’t decipher much of what’s being said here [...]

I've restructured this so it (hopefully) much more clearly explain what's not supported. Support is much improved since when you wrote these comments, and we now support all actual use I'm aware of (except for your data using TAPE=SS with clinos and heights in Mammoth, which you concluded was all incorrect anyway).

The changes are in 24ab9ca5b7efba35689e015cc3df59cda08d830f. You can view a formatted version of the latest version on github at ​https://github.com/ojwb/survex/blob/main/doc/walls.rst (that's with github's restructured text handling, so the formatting may differ a little compared to the manual as built by our source tree).

[Joe Walko]

The updated Walls chapter looks really good, imo. Looks like you put a quite a bit of effort into that rewrite.

A couple things regarding diffpos:

• I might have missed it, but I couldn't find any mention of diffpos output column order being x, y, z. I had to dig a bit to confirm my assumption of x/y/z order.
• I knew it was meters, but that probably should be made clear as well, given that diffpos accepts a Walls coordinate listing in feet, which img converts to meters.

Also, your comments (below), or something similar, are worth noting in the Walls chapter. Being able to "*include" Walls SRV files via Survex SVX file is non-obvious to the Survex beginner. You cover this effectively in the Compass chapter, but not the Walls chapter.

However you could migrate to using Survex by replacing the .wpj with a simple .svx file which includes your .srv files using e.g. *include file1.srv. (If you're setting options in the .wpj you'd need to copy those to #units at the start of each .srv.)

Then going forwards, you can either add new data as .srv or .svx.

[Olly Betts]

Replying to Joe Walko:

The updated Walls chapter looks really good, imo. Looks like you put a quite a bit of effort into that rewrite.

A couple things regarding diffpos:

• I might have missed it, but I couldn't find any mention of diffpos output column order being x, y, z. I had to dig a bit to confirm my assumption of x/y/z order.

Hmm yes, it's kind of hinted at by the diffpos section but should be explicit. I think more useful is to clearly state it's (East,North,Up) actually, so I've done that in the diffpos section and in the Walls chapter.

• I knew it was meters, but that probably should be made clear as well, given that diffpos accepts a Walls coordinate listing in feet, which img converts to meters.

That too.

Also, your comments (below), or something similar, are worth noting in the Walls chapter. Being able to "*include" Walls SRV files via Survex SVX file is non-obvious to the Survex beginner. You cover this effectively in the Compass chapter, but not the Walls chapter.

However you could migrate to using Survex by replacing the .wpj with a simple .svx file which includes your .srv files using e.g. *include file1.srv. (If you're setting options in the .wpj you'd need to copy those to #units at the start of each .srv.)

Then going forwards, you can either add new data as .srv or .svx.

Also done.