Survex excludes Walls shots from network with variance of (*) or (x,?)

[Joe Walko]

Docs indicate line of Walls shot data floated via (*), or floated vertically via (x,?) should should be treated as 'nosurvey', yet still part of the network (unless I'm misinterpreting the documentation).

Instead, Survex excludes affected stations from network, and issues warning: as: cavern: warning: Survey not all connected to control points The following survey stations are not attached to a control point:

Behavior is sometimes inconsistent, and non-obvious: A downstream float tie can cause the above error for a station several hops removed from the (*).

[Joe Walko]

Three sets of Walls test data, Walls coordinate lst, and cavern logs

[Olly Betts]

Survex's nosurvey style only indicates that two stations have a line-of-sight connection (or something approximating one). It's an infinitely elastic leg which has no inherent vector of its own and requires both ends to be positioned by being connected to fixed points by other survey legs.

Currently we map Walls' (?) and similar things to Survex's nosurvey, but it's not a perfect match. AIUI Walls' version is infinitely elastic but has its own vector which is used if one end is not otherwise positioned. Survex doesn't currently have a way to support that exactly. So this mapping gives the correct result if both ends are positioned, but a hanging survey if they aren't.

For horizontal-only and vertical-only connections we're meant to instead map to a leg with a standard deviation of 1000m in the decoupled directions (your test3.srv shows this is buggy - I have a candidate fix and your survex_test3.wpj now seems to give the expected result, so I'll push a fix once I've added a regression test to the testsuite). This should give something close to Walls semantics, but the leg can still have some influence in the direction which is meant to be decoupled. Proper support for this in Survex is something people want (#124) but I haven't implemented it yet. Once that's done, we should be able to support horizontal-only and vertical-only connections in Walls properly.

We could instead map (?) to a leg with large standard deviations in all directions, but this has more downsides if people don't actually make use of such legs having their own vector so I went for mapping to nosurvey. We could change that.

Actually supporting (?) with Walls' semantics is probably feasible once #124 is addressed, but seems like it's going to need additional work as well.

I doubt I'm going to attempt to fully support Walls semantics for floating the traverse containing a leg with (*); currently we treat that like (?). Maybe we should also warn, but I dislike warnings which there isn't a good way to suppress; changing the data to use (?) would avoid the warning but might be undesirable. Maybe you can provide insight as to what people expect when they use (*) rather than (?)?

[Olly Betts]

Replying to Olly Betts:

I have a candidate fix and your survex_test3.wpj now seems to give the expected result, so I'll push a fix once I've added a regression test to the testsuite

Fix pushed, will be in 1.4.23: 782b28470dab1a5bf88e606e7fa8624104482176

[Olly Betts]

Behavior is sometimes inconsistent, and non-obvious: A downstream float tie can cause the above error for a station several hops removed from the (*).

When there's a hanging survey, Survex doesn't report every unconnected station as that could be hundreds or thousands of stations. Instead it picks one in each "chunk" of hanging survey to report (currently it can actually report more than one per chunk in some cases, but that's really a bug). It's not an entirely arbitrary choice - it should be a junction, and shouldn't be an anonymous station. So it's unlikely to be the station at the unconnected end of a (*) tie - it would have to be a junction ignoring the (*) tie, and that be the junction that's picked.

I can see it would be more helpful to point the finger at the station next to the floated leg in this case, but that'd be rather more complicated to achieve than the current algorithm, and it's specific to how Survex currently handles floating legs in Walls data. I think it makes more sense to put the effort into improving the interpretation of floated legs so they no longer result in hanging surveys.

[Olly Betts]

(currently it can actually report more than one per chunk in some cases, but that's really a bug)

That bug is actually easy to fix thanks to other changes in this area over the last few years, so I've pushed a change that means we should now report exactly one station per hanging component in the survey. We could also actually fail to report a station (the situation is obscure and requires a lot of overlapping loops but with inside knowledge I managed to construct an artificial test case for it on the first try).

Anyway, that means we can now improve the wording here.

[Joe Walko]

I had misremembered the meaning of (*) versus (?). Perhaps I should stop using carbide lights for all the interior lighting in my house - the fumes seem to be affecting me mentally.

I had been thinking that (*) was simply a shortcut to writing (?,?). This is woefully incorrected. Fortunately, I was able to restore a correct version of my corrupted memory from offline backup.

Variance override of * affects the entire traverse whereas '?' is specific to the single line of shot data. What constitutes an "entire traverse" does not get determined until data-compile, in Walls, and is thus can be somewhat opaque to the user when working on data.

I am now questioning all cases where "*" was used for per-shot variance-override in the data I help with, and am in the process of evaluating each occurrence.

I am also pondering whether or not we should be using a 'very large number' instead of "?" when floating a single shot. Which is what we do already at the SRV level with UV/UVH/UVV=999999 when the intent to to float the entire srv, as UVx does not accept "?" or "*" as arguments.

#UNITS UVx and per-shot variance overrides are not the same thing, of course: the former being a 'scaling' factor, the latter representing a specific length or RMS value, if it starts with "R" or "r".

[Olly Betts]

I'd guess the "entire traverse" is found by following connected legs from the (*) marked leg in both directions until we reach a station which doesn't have exactly two connected legs.

It has some corner cases though, like does it stop when it finds a leg with a different variance override or does it keep going? If it keeps going, does the different variance override beat the (*)?

I haven't implemented UV, etc yet. They aren't used in any of the Walls datasets I have access to, and there isn't an existing variance scaling mechanism in Survex to just hook them up to, plus Survex uses a different error model to Walls so station positions won't be completely identical anyway. So they seemed a low priority but if they're actually used and especially if they're used with really large values to pseudo-float data that probably bumps the priority.

It might make sense to implement proper support for (?)/(?,x)/(x,?) first and then treat UV factors above some reasonable threshold as infinity.

[Olly Betts]

82c7ba54dd401b19faf1b3d875bdb597e8e8019c adds support for UV/UVH/UVV. I have no examples which use them, so they've not had much testing.

They currently scale the covariances too, which seems like the right approach except it seems it might have unwanted effects when using a large factor to approximate floating. I guess we'll see if this actually causes problems on real data.

[Joe Walko]

vo (*) image 1

[Joe Walko]

vo (*) image 2

[Joe Walko]

Variance overrides of (*)/(*,)/(,*)/(?)/(?,)/(,?)/(*,*)/etc seem work much better now.

I believe there are some edge-cases remaining with (*), but they're rare. See the two images attached for a couple of examples. Not enough info there for you to debug, I realize; but otherwise I'll loose track.

I think this can be closed. When I have some time, I'll try to isolate the specifics of these 'edge-cases', and will open new tickets for those specific cases, along with some useable sample SRV data.

[Olly Betts]

Both the images look like the situation I described above:

Currently we map Walls' (?) and similar things to Survex's nosurvey, but it's not a perfect match. AIUI Walls' version is infinitely elastic but has its own vector which is used if one end is not otherwise positioned. Survex doesn't currently have a way to support that exactly. So this mapping gives the correct result if both ends are positioned, but a hanging survey if they aren't.

(Here "similar things" would cover (*) and (?,?) where the floating is in all directions, but not e.g. (?,4).)

We could instead map (?) to a leg with large standard deviations in all directions, but this has more downsides if people don't actually make use of such legs having their own vector so I went for mapping to nosurvey. We could change that.

Using nosurvey should very closely match Walls semantics provided there's a connection from the station to a fixed point which doesn't have a leg mapped to nosurvey, but it results in dangling survey data if the only connections from it to fixed points are nosurvey.

Large standard deviations handles data only connected by (?) legs, but is a less precise interpretation of the data in cases which work with the current approach.

Picking which to use based on checking connections would be fiddly because we haven't seen all the data at the point where we handle this.

So it's a trade-off essentially. I picked nosurvey suspecting that the situation might not occur in practice, but given it does I think the second approach makes more sense.

Actually supporting (?) with Walls' semantics is probably feasible once #124 is addressed, but seems like it's going to need additional work as well.

And this should provide a better solution in the longer term.

[Olly Betts]

Replying to Olly Betts:

We could instead map (?) to a leg with large standard deviations in all directions, but this has more downsides if people don't actually make use of such legs having their own vector so I went for mapping to nosurvey. We could change that.

Using nosurvey should very closely match Walls semantics provided there's a connection from the station to a fixed point which doesn't have a leg mapped to nosurvey, but it results in dangling survey data if the only connections from it to fixed points are nosurvey.

Large standard deviations handles data only connected by (?) legs, but is a less precise interpretation of the data in cases which work with the current approach.

Picking which to use based on checking connections would be fiddly because we haven't seen all the data at the point where we handle this.

So it's a trade-off essentially. I picked nosurvey suspecting that the situation might not occur in practice, but given it does I think the second approach makes more sense.

I've changed this in 97ef7ad3fad790bbeaddd8a9a168ce5b9ccfe272.

4e6b965fa6ea1f3b6d5483d4a5164eee9de88e12 fixes a related problem - Survex was just ignoring a floated #FIX, e.g.:

#FIX A 0 0 0 (?)

However testing in Walls shows this should still fix A if there are no other connections to fixed points, and now it does.

[Joe Walko]

Variance-override of "(*)" remains problematic. In the large data, most all of the surveys with major coordinate discrepancies between Walls & Survex are those with a "traverse override" with "(*)" on one more lines of shot-data.

Variance override of (*) basically has no effect in Survex - affected stations end up with the same coordinates as equivalent shots with no variance override or UV adjustment.

In Walls, (*) roughly places the stations close to where they would be if I use UV=1000 for the section, or variance override of (1000) or variance override of (?) for each individual line of shot data.

The rub is that (*) applies to the "entire traverse", hence replacing the (*) with high UV for the section or per-line variance-override is not always straightforward.

Ostensibly, if we know a traverse is "bad", then we really ought to explicitly float each shot imo, but I'm not sure I can get all four data managers to concur. Three will concede, one of them will push back.

[Joe Walko]

Walls data illustrating issues with variance-override of (*). Look at "VO_AST:A2" coordinates in Survex vs Walls.

[Olly Betts]

Variance override of (*) basically has no effect in Survex - affected stations end up with the same coordinates as equivalent shots with no variance override or UV adjustment.

Currently (*) should be handled by Survex as if you'd written (?) (i.e. float just one leg instead of the whole traverse).

The first problem with implementing (*) are that it's not clear what "entire traverse" even means. The Walls documentation seems to hint that a traverse is defined like in Survex and so we should follow connected legs each way until we get to a junction or dead end, but then connecting a new survey in the middle of a (*) floated traverse will suddenly mean the part of it on the other side of the new connection to the leg marked with (*) isn't floated. Is that really how it works? If so, that seems a strong argument for explicitly marking all the legs you want floated with (?) since then future connections won't change what gets floated.

How (*) and other overrides interact is unclear to me - if other legs in the traverse have their own overrides, what happens?

It also seems like it'd be a reasonable amount of work to implement. We'd need to store a list of all the (*) marked legs and process them just before we solve since we can't determine the traverses until then, but we need to update the variances before then. And this work is just to implement a Walls feature that seems to have quite limited up-take. I also can't see the code being usefully reusable (unlike adding support infinite variances which is something people have asked for native Survex data as well).

Maybe I'll try to implement (*) at some point, but other things are a higher priority.

[Joe Walko]

After the previous posting, I contemplated this some more, and have concluded that VO of (*) is not something Survex needs to support. There is too much ambiguity here, and, as you noted, limited uptake. Additionally, I'm finding cases where folks used (*) when their intent was to float only the shot, not the entire traverse.

Posting a "warning" and treating the "*" like "?" is appropriate Survex behavior, imo. This deviates from Walls' behavior, but the nebulous use of (*) to float the entire "traverse" seems like a bad practice - folks should float each leg, or pseudo-float the section via a high UV value in #UNITS.

The functionality in Walls is no doubt a side-effect of the UI options where one can manually float traverses. Those UI "floats" get stored only in the work files, and are not fully persistent. Th VO override of (*) no doubt evolved as a way to persistently encode those UI floats into the data.

I've examined all occurrences of (*), (*,) and (,*) in the large dataset, and they can all by changed to not use (*) with minimal effort.

[Olly Betts]

The functionality in Walls is no doubt a side-effect of the UI options where one can manually float traverses. Those UI "floats" get stored only in the work files, and are not fully persistent.

Aha, so it was probably an interactive feature and once the mechanism existed it then got exposed in the survey data syntax (or perhaps it was exposed as a way to be able to write automated tests). In interactive use, there isn't the problem of the legs in the affected "traverse" potentially changing as new data is added. Also the user can presumably get some sort of visual feedback as to what is getting floated.

Posting a "warning" and treating the "*" like "?" is appropriate Survex behavior, imo. This deviates from Walls' behavior, but the nebulous use of (*) to float the entire "traverse" seems like a bad practice - folks should float each leg, or pseudo-float the section via a high UV value in #UNITS.

My only qualm about a warning here is that addressing the warning could require some fiddly changes to the Walls data that someone might not want to make. This is mitigated by (*) not being used much, but it is potentially very annoying if you were using it.

I've wondered if we could (ab)use macros to provide an explicit warning suppress mechanism, e.g.:

1   2   12.34 345 +67 $(survex_no_warn_float_all) (*)

With that macro defined as empty, and Survex suppressing the warning for the rest of the line when it sees a macro with that particular name expanded. However I think macros don't work in vector data (or #fix IIRC), so this idea would probably only work as a way to suppress warnings in directives.

A similar idea but with a magic comment could work (we'd need to scan the rest of the line when we want to emit the warning, but that's totally feasible):

1   2   12.34 345 +67 (*) ; survex::no_warn_float_all

I'll ponder - maybe there's a neater way we could allow suppression of the warning. It does seem like something which it is useful to warn people about.

[Joe Walko]

Walls macros can only be used as arguments to units parameters, hence their usefulness is limited. They'd likely need to go in WPJ .OPTIONS, which is a feature the typical Walls user is oblivious to.

I'd be interested in Survex "directives" of some sort which could be used do things like your "survex_no_warn_float_all" example, but "globally". These would go into a .svx wrapper/container, followed by *include of the WPJ file(s). Perhaps defined via *set, or something similar. Granted, one loses granularity as these would have affect the entire WPJ project. But they could perhaps be supplemented by magic comments.