Walls data: Explicit null prefix2 and prefix3 in shot data station names does not properly override #PREFIX2 and #PREFIX3 directives

[Joe Walko]

If prefix/prefix2/prefix3 is set via #PREFIX/#PREFIX2/PREFIX3 directive, cavern does not override #PREFIXn directive with blank prefix when a station name contains explicitly blank prefix/prefix2/prefix3.

The attached SRV file shows it better than I can describe it. The SRV contains Walls test data, and, in comments, station listing from Walls, station list from dump3d, and relevant 'warning' messages from cavern log file.

---

I noticed some station names were not matching up between survex and Walls for the large dataset I've been testing with. PREFIX2/PREFIX3 are yet another somewhat esoteric feature we use; I assign unique PREFIX2 to each of four datasets when merging into one, hence most every station has PREFIX2. along with limited use of PREFIX3. There are certain cases where we have explicitly blank prefix names as part of the station names, thus causing the station name mismatches.

[Joe Walko]

Sample Walls data showing explicit prefix issue; diagnostic walls and cavern info embedded as comments in srv

[Olly Betts]

We map Walls prefix levels into Survex's prefix levels, but Survex doesn't allow an empty prefix level so we currently "squash away" empty levels. I'd thought Eric said this didn't look like it would cause problems in practice, but it seems it does.

We handle an empty station name by replacing it with (literally) empty name, which contains a space so cannot collide with a "real" Walls station name. E.g. PEP: becomes PEP:empty name. I didn't find anything saying the name could be empty, though also nothing which clearly says it can't be, but Walls allows it in practice.

We can handle empty prefix levels similarly. Probably we want something shorter than empty prefix though since empty prefix levels are actually documented as a feature, and seem less weird than an empty station name, so they're probably more common. Also there can be two empty prefix levels which would result in e.g. X:::1 -> X:empty prefix:empty prefix:1.

The manual implies the same characters are allowed in a prefix as a station name, and the documented disallowed characters in a station name are:

• : - but this is the prefix level separator so we can't use it here
• ; too visually similar to : I think - consider X:;:;:1
• ,
• '#' (but in fact this is allowed in station names)
• tab
• space

I think the least terrible option is to use a single space (which doesn't work for an empty station name as it's invisible at the end of a name when displayed, but a single space for an empty prefix level will always be between two :).

We can omit any leading empty prefix levels without ambiguity (like Walls does), so your stations would be:

:::d1                        -> d1                           (same as Walls)
Prefix3::shotpref:d1         -> Prefix3: :shotpref:d1
Prefix3:Prefix2:shotpref:d1  -> Prefix3:Prefix2:shotpref:d1  (same as Walls)
Prefix3:shotpref::d1         -> Prefix3:shotpref: :d1
::shotpref:d1                -> shotpref:d1                  (same as Walls)
:shotpref::d1                -> shotpref: :d1
shotpref:::d1                -> shotpref: : :d1

And examples with empty station names:

::PEP:    -> PEP:empty name
:PEP::    -> PEP: :empty name
PEP:::    -> PEP: : :empty name

[Olly Betts]

Only "Unused fixed point" warnings remain, and the output seems to be consistent with Walls:

NODE 586700.00 4116200.00 300.00 [shotpref: : :d1] EXPORTED FIXED
NODE 586700.00 4126300.00 300.00 [Prefix3:shotpref: :d1] EXPORTED FIXED
NODE 586700.00 4136400.00 300.00 [Prefix3:Prefix2:shotpref:d1] EXPORTED FIXED
NODE 586700.00 4136500.00 300.00 [Prefix3: :shotpref:d1] EXPORTED FIXED
NODE 586700.00 4136500.00 300.00 [shotpref:d1] EXPORTED FIXED
NODE 586700.00 4136500.00 300.00 [shotpref: :d1] EXPORTED FIXED
NODE 586700.00 4136500.00 300.00 [d1] EXPORTED FIXED

There's another wrinkle in this area though.

Consider:

#fix :::X  0 0 0
#fix ::X:Y 1 1 1

Currently those map to X and X:Y in Survex, but that means that X is both a station and a survey in Survex's name hierarchy, which is not allowed.

I don't currently have an idea I like for addressing this. A key problem is that it probably rarely occurs in practice so we really don't want to change name mapping of Walls stations in ways that affect common use patterns (e.g. we could map :::X to, say, empty 3:X but that would affect the names of all stations with leading empty prefix levels).

There's a secondary problem with different prefix levels getting conflated - in your example ::shotpref:d1, :shotpref::d1 and shotpref:::d1 all become a name with the first component shotpref, but that's a different shotpref in each case! Survex uses the names for surveys, so you might have cave.area.survey.1 and tools working with Survex names will typically assume such a naming scheme, which might not work well here. I don't really know how Walls users use the prefix levels in practice - perhaps because of the convention of omitting leading empty prefix level these would be regarded as the same shotpref by a Walls user.

[Olly Betts]

Reported problem fixed by 14a55b3cd152b5cb5e9752e6ddea292ae15111c1.

[Joe Walko]

I suspect the answer to "how Walls users use the prefix levels in practice", is mostly "they don't", beyond first-level prefix.

My most common use-case is using a different PREFIX2 for each of the four datasets, which is inserted as a 'compile option' for that dataset's branch when I run the 'merge'. Thus, everything in each dataset gets its own prefix2, post-merge; no changes were needed to the individual datasets for this to work, since those don't use prefix2 or conventions such as ::station-name.

Within each dataset's branch, fixed stations are commonly defined with no PREFIX1 (aka PREFIX), indicating dataset-wide relevance.

After the merge, such stations, now having prefix2 but explicit empty prefix1, those stations look like [PREFIX2::FixedStationName] in Walls on onscreen maps and exports, if one selects option to display prefixes.

For now, in Cavern, excluding the 'other wrinkles' situations, this seem to be working OK, resulting in name such as [PREFIX2: :FixedStationName], with space character filling in for prefix1.

[Olly Betts]

If I follow correctly then stations will generally always have the same highest level prefix set, except in the (fairly common) case of a single dataset using only one prefix level and omitting the prefix for fixed points. So:

• different prefix levels being conflated won't happen in practice
• the "same name for station and survey" wrinkle would only happen for a single dataset and then requires a fixed point name coincides with a prefix name (e.g. :::X1 and ::X1:AA1)

[Joe Walko]

Testing with my merged dataset which uses both PREFIX2 as well as empty PREFIX1 indicates this is resolved. Using <space> in survex for empty intermediary prefixes is a suitable workaround, and easily stripped out in downstream apps in cases where one needs to perform a literal comparison between Walls exported station names and survex exported station names.

Lists of all stations exported from Walls and Survex now agree wrt their names (excluding the extra space for empty intermediary prefix); previously they did not.

This can be closed. I'll open new ticket if I run into new but related Prefix issues.

As an aside, Walls has it's own prefix-related issues, albeit in secondary functions. Shapefile exports, for example only allow 8-characters for the combined prefix string - not long enough if one is using multiple prefix levels (I've patched that in version I run; I need push that to Andy).

[Olly Betts]

easily stripped out in downstream apps in cases where one needs to perform a literal comparison between Walls exported station names and survex exported station names

Indeed - you can just remove any spaces since they aren't valid in Walls station names or prefixes. I'll note that in the documentation.

I should also add a note about the "same name for station and survey" wrinkle. It's not likely to affect many users, but I'm going to split out a section for obscure/esoteric details thanks to your manual comments and it can go there.

[Olly Betts]

Replying to Olly Betts:

easily stripped out in downstream apps in cases where one needs to perform a literal comparison between Walls exported station names and survex exported station names

Indeed - you can just remove any spaces since they aren't valid in Walls station names or prefixes. I'll note that in the documentation.

We were actually missing a section on empty prefix levels, so I've added one and noted how to map back to Walls names in 3c3e04a340927dde86b30414fe40253e6227cc08.

I should also add a note about the "same name for station and survey" wrinkle. It's not likely to affect many users, but I'm going to split out a section for obscure/esoteric details thanks to your manual comments and it can go there.

Done in e6d8d6fde1dc589f46b03760d130fad4164dbe0d.