Bugs in UV=/UVH=/UVV= handling

[Joe Walko]

###Notes on milestone v1.4.23, commit d895951a

###Previously I did not have handy Linux build environment to build from source; that's been rectified.

###Notes below reflect testing on Ubuntu 24.04 enviroment, survex milestone v1.4.23, commit d895951a. Prior feedback was from Windows 10, Survex release 1.4.22.

---

###Something new. UV or variance related, I presume:

Fatal error:

Simplifying network... not sym - 1,0 = 22531.619363, 0,1 was 22531.619363 netbits.c:703: smulvs didn't produce a sym mx cavern: error: Bug in program detected! Please report this to the authors

### I narrowed this one to just one of the four datasets which get merged, and unsurprisingly, it's the dataset which has prolific usage of UV/UVH/UVV and variance overrides

### I went through and tried to comment out instances where UV/UVH/UVV is set to a very-high-number, or is set to "0", but I'm sure I missed some. Error changed a bit after commenting those out:

Calculating traverses... netskel.c:666: assert(!fEquate) failed cavern: error: Bug in program detected! Please report this to the authors

### At this point I didn't pursue any further, but I assume the error is UVx or variance related.

---

###Massive decrease in quantity of frontsight/backsight disagree warnings; this effectively resolves the issue with warning logfile clutter.

---

###Explicit empty prefixes seem to be bahaving better, albeit with a 'space', which seems OK for now ; see additional comments in ticket #163.

---

### IH/TH data gets discarded for underwater surveys if TAPE=SS and ORDER=DAV. But this combination is valid, since such surveys typically start or end above water, and often have some DAV shots, with inclination instead of IT/HT

CFSB1382.SRV:17:16: warning: Instrument and target heights currently ignored with Walls option combination “TAPE=SS” and “ORDER=DAV”

S1 S2 70 84 -- 0 15 *8 8 0 5*

^~

CFSB1382.SRV:18:16: warning: Instrument and target heights currently ignored with Walls option combination “TAPE=SS” and “ORDER=DAV”

S2 S3 60 89 -- 15 31

CFSB1382.SRV:19:16: warning: Instrument and target heights currently ignored with Walls option combination “TAPE=SS” and “ORDER=DAV”

S3 S4 15 84 -- 31 34

CFSB1382.SRV:20:16: warning: Instrument and target heights currently ignored with Walls option combination “TAPE=SS” and “ORDER=DAV”

S4 S5 22 82 -- 34 33

---

###Comma in variance override probably shoudn't trigger a warning, since (UVH, UVV) form isn't all that unexpected.

CFSB1253.SRV:46:50: warning: Interpreting “,” as separating readings but may be intended as a decimal point

C1252:T20 F28 9.3 246 -28.75 -- -- *0 10 1 .5* (3,3)

[Olly Betts]

Simplifying network... not sym - 1,0 = 22531.619363, 0,1 was 22531.619363 netbits.c:703: smulvs didn't produce a sym mx cavern: error: Bug in program detected! Please report this to the authors

I'd guess this is a very large UV= or similar value causing us to lose accuracy on the covariances (they match to the 11 SF shown at least).

It'd be very helpful to have a reproducer for this. What are the larger numbers you have here?

Calculating traverses... netskel.c:666: assert(!fEquate) failed cavern: error: Bug in program detected! Please report this to the authors

I suspect that's mishandling UV=0 or similar.

### IH/TH data gets discarded for underwater surveys if TAPE=SS and ORDER=DAV. But this combination is valid, since such surveys typically start or end above water, and often have some DAV shots, with inclination instead of IT/HT

I've pushed 823c9310cd6000d0c402a6e2e5b9d4855d22703d which maps such legs with IH/TH and omitted clino to diving data. You still can't have both clino and IH/TH on the same leg with TAPE=SS though.

CFSB1253.SRV:46:50: warning: Interpreting “,” as separating readings but may be intended as a decimal point

C1252:T20 F28 9.3 246 -28.75 -- -- *0 10 1 .5* (3,3)

Fixed by 0ce52ab3a891725c40045c97ac4443668ed15d00.

[Olly Betts]

The remaining issues here are just the UV= ones, so changing this to track those.

[Olly Betts]

Previously I did not have handy Linux build environment to build from source; that's been rectified.

I've just added a new feature you might find useful, since you can now build from git.

I got annoyed at the manual steps involved in comparing the results of Walls and Survex processing, so I've added support for reading Walls .LST files to the "img" library, which means Survex can now transparently read them as if they were processed survey data (like a .3d file or a Compass .plt file).

This means you can now easily find any stations which are in only one of the datasets or whose coordinate differ by more than a specified threshold along any axis - e.g. to allow up to 50cm difference:

diffpos SURVEY.LST SURVEY.3d 0.5

If you select Connected vectors with file references when you create it, the .LST file actually says which stations connect to one another, and aven SURVEY.LST will show the survey centreline.

[Olly Betts]

Replying to Olly Betts:

Calculating traverses... netskel.c:666: assert(!fEquate) failed cavern: error: Bug in program detected! Please report this to the authors

I suspect that's mishandling UV=0 or similar.

This trivial example

#UNITS UV=0
A1     X1      12.34   065     0

triggers an assertion, though not exactly the same one:

netskel.c:855: assert(!fZeros(&leg->v)) failed
/home/olly/git/survex/tests/../src/cavern: error: Bug in program detected! Please report this to the authors

I've addressed it by using a very small but non-zero scale factor instead for now (commit 17c737a436579e4f5846b99e8118ce4055204203). I'm reasonably confident it'll fix the one you reported above too, but please retest.

(I've not looked at very large UV= values yet.)

[Olly Betts]

though not exactly the same one:

Aha! If I create a loop with one leg UV=0 I could trigger the assertion you did before, and it is fixed now.

[Joe Walko]

tl;dr: UV=0 issue does indeed seem resolved. Wrt to large UV values: Cavern is happy if I use UV=9999 for all instances where I have UV=<large-number> But crashes if I use UV=99999.

---

The update resolved the "second" error, which I started seeing after stripping cases where UV=0. (Partial .3d file gets written in this case).

Now I'm back to the earlier initial error: Simplifying network... not sym - 1,0 = 2253.154644, 0,1 was 2253.154644 netbits.c:703: smulvs didn't produce a sym mx (No partial .3d file, in this case).

We have various instances where UV or UVV=999|9999|99999|999999. Pseudo-floating all part of an SRV. It's likely the '999...' permutations are not intentional, as opposed to a 'standard' very-large-number.

I replaced all those '999...' permutations with UV=$(FLOAT) (or UVV=) macro so I could more easily test different values by adjusting value via the macro definition in the project-wide compile options.

Cavern is happy with the with '9999' for the large UV value ($FLOAT=9999 in compile options).

But not with '99999', which results in an abort: Simplifying network... not sym - 1,0 = 2253.154644, 0,1 was 2253.154644 netbits.c:703: smulvs didn't produce a sym mx

No partial .3d file created.

I didn't have to time to do additional isolation or testing.

Possibly easily resolved by by capping the UV value, internally in cavern.

[Olly Betts]

Possibly easily resolved by by capping the UV value, internally in cavern.

I'll do that as capping is much better than just ignoring UV=, etc, as the current released version does (also better than crashing with a failed assertion!)

Once Survex supports infinite variances we can treat scale factors above a threshold as infinite, which is actually what the user wants in these cases.

It'd be good to investigate exactly what's happening here, but without a reproducer it's hard to. It'll likely need a survey sufficiently complex that we have to actually solve a matrix, but also with particular parts pseudo-floated with UV=999999 or similar. It may just be that some combinations of high scale factors from UV= and high (co-)variances from variance overrides means we exceed the precision of double, but it's possible some calculations are being done in the wrong order such that they needlessly suffer from lack of precision.

[Olly Betts]

Cap added in 3e08f0302ba37b25cc98736c9205c75fcfe611f6.

[Joe Walko]

Confirming this is resolved per testing with my large dataset.

The one dataset manager (of the four) who is the primary user of UV=<very-large-number> confirms that his choice of 999/9999/99999/999999/9999999 is random - he's not trying implement multiple levels of pseudo-floating.

There's no reason he needs to use anything larger than UV=9999, hence capping in survex doesn't create an issue here. Meanwhile, I'll have him standard his usage to 9999.

Separately, I'll likely implement a project-wide UV=($FLOAT) macro for UV/UVH/UVV pseudeo-floating.