Attributes in shapefile exports

[Eric C. Landgraf]

Walls includes several attributes in its shapefile exports. For example, the line export includes from and to station names, the X, Y, and Z of the segment, the computed length/azimuth/clino of the segment, and other metadata (date, etc. Bascially anything in the equivalent of a .3d file).

It would be helpful if Survex included equivalent attributes, or even richer data in shapefile or other georeferenced outputs. ("other" suggested because: ​http://switchfromshapefile.org/)

[Eric C. Landgraf]

shapefile attributes from walls

[Olly Betts]

What's "LINETYPE"?

[Eric C. Landgraf]

I took a look, and all the values are a Hex code. I assume it's walls' internal represenation of their colouring, line width, and dash pattern. i.e. probably not very useful or well-used.

[Olly Betts]

OK, and I assume CTR must be "CenTRe". The data we currently have for each leg is:

• the 3D vector after loop closure
• from and to stations
• survey the leg is in
• surveyed date of that survey (which can be a range)
• style (normal/diving/nosurvey/etc)
• surface flag
• duplicate flag
• splay flag
• various closure error info

Each station has a name, 3D position, and some flags: surface underground entrance exported fixed anon wall

[Eric C. Landgraf]

yes, that is correct. so CTR_Z is the elevation of the center of the leg.

Flags and styles would be useful to filter by. e.g. I might want a separate display style for surface lines, and want to filter out all splay legs.

[Joe Walko]

This functionality is crucial to those of use who incorporate the post-processed cave data into GIS software. Metadata included by Walls and Compass in the shapefile exports is used for symbology (color by date, for example), labeling, categorizing, and creating subsets, in GIS, of cave data based on certain attributes.

In addition to GIS, the shapefile attribute metadata is used for downstream analysis, and as input to apps one can use for generating and viewing 3D imagery - imagery where it's important to retain key metadata.

This metadata is also import for validation and analysis of post-processed data. One can easily see the final azimuth of any shot in Compass or Walls shapefile exports. That's far more difficult with Survex 3d output which is xyz coordinate based. Makes it hard to compare final shot azimuths without extra post-processing of the 3d data.

[Joe Walko]

LINETYPE is described in the Walls manual, 'Exporting Shapefiles' chapter, as "RRGGBBxx (RGB color/style)". RBG is the color selected in the Walls 'Segment" UI tab. The 'xx' style value maps back to the 'line style' one selects in the "Line style" one selects, where there are seven options - those 'line styles' are represented as 0 through 6 in the shapefile LINETYPE field. tab.

LINETYPE field in shapefile export is likely not used by anyone. Except for me :-). I assign a specific line-type style to all shots which are to be excluded from length, via Walls UI, then in post-processing, I leverage that to calculate accurate statistics. Needed only because Walls does not otherwise provide a rational way to exclude designated shots from total length.

In general, one would not ever depend on LINETYPE field in shapefile - that field is an anomaly in that it is based on user-actions via the Walls UI, and is not set in underlying WPJ or SRV data. It's semi-transient, and gets reset if one purges workfiles, in Walls.

[Joe Walko]

Both Compass and Walls include rich attribute metadata in their shapefile exports. This approach is similar, but available fields do vary. Both should be looked at here, not just the field Walls populates in its shapefile exports. Compass has fields which Walls lacks, and vice-versa.

McKenzie documents the Walls shapefile fields in the Walls manual - but I'm not seeing Compass documentation covering their shapefile fields. I can pull together something describing Compass shapefile fields if it'd be useful.

[Joe Walko]

Rudimentary info on attribute fields which COMPASS includes in exported 'shapefile' sets. Scope and approach is similar to that of Walls.

Attribute fields included in Compass-exported "shot" based shapefile sets, including:
3Dshots/2Dshots, 3Dpass/2Dpass, 3Dsplt/2Dsplit, 3Dmprism.shp, and 3Dprism.shp:

SECTION	String	(80.0)
SURVEY	String	(20.0)
MONTH	String	(2.0)
DAY	String	(2.0)
YEAR	String	(4.0)
FR_STATION	String	(20.0)
TO_STATION	String	(20.0)
ELEVATION	Real	(4.2)
AZIMUTH	Real	(4.2)
INCLINATION	Real	(4.2)
LENGTH	Real	(4.2)
COMMENT	String	(160.0)
FACE	Real	(4.2)*

*('FACE' only present in 3D passage imagery shapefiles)

Attribute fields included in Compass-exported "3Dstat/2Dsta" aka "stations" shapefile sets:

SECTION	String	(80.0)
STATION	String	(20.0)
COMPASS_X	Real	(18.5)
COMPASS_Y	Real	(18.5)
COMPASS_Z	Real	(18.5)
LEFT	Real	(4.2)
RIGHT	Real	(4.2)
UP	Real	(4.2)
DOWN	Real	(4.2)
COMMENT	String	(160.0)

Attribute fields included in Compass-exported "3Dsrv/2Dsrv" aka "surveys" shapefile sets:

SECTION	String	(80.0)
SURVEY	String	(20.0)
MONTH	String	(2.0)
DAY	String	(2.0)
YEAR	String	(4.0)
COMPASS_NMI	Real	(18.5)
COMPASS_NMA	Real	(18.5)
COMPASS_EMI	Real	(18.5)
COMPASS_EMA	Real	(18.5)
COMPASS_VMI	Real	(18.5)
COMPASS_VMA	Real	(18.5)
COMMENT	String	(255.0)

No attribute data is exported in 3Dbound or 3dmbound shapefile sets.

[Olly Betts]

Thanks.

It seems really unhelpful to export the date as three separate fields. Not sure if there's a specific date type available here but if not an integer count of some time units since some epoch date would allow for filtering based on date without having to jump through hoops.

What are COMPASS_NMI, etc? The COMPASS_ part is presumably the name of the software rather than the instrument, and N/E/V are presumably the 3 axes. Are MI and MA "min" and "max"? So do these give an X/Y/Z aligned bounding box for each survey?

[Joe Walko]

Attached zip contains sample Compass and Walls shapefile exports, along with the Compass and Walls cave data. Same small cave for each, but there may be minor differences from when I converted from one format to another (back in 2024, hence I don't recall the process I used).

---

Walls shapefiles are included 'as is'. Walls shapefile exports have a few limitations, which I'll eventually address (already fixed in the bespoke version I run). Including: Only 8-characters for "prefix3:prefix2:prefix" field, insufficient precision, all LRUDs for station get skipped if any of them are 'empty', and inconsistent meters vs feet for LRUD lengths.

---

For Compass, I excluded some redundant sets of shapefiles:

Only Compass shapefiles *srv, *sta, *shots, *pas, *mprism and *prism contain attribute data, which is stored in the .DBF file. Thus, other shapefile sets are excluded here.

*_3Dshots, *_3Dpas, *_3Dmprism, *_3Dprism all have identical attribute data in DFB files, except prism and mprism file duplicate the shot-data eight times, and have FACE field, which contains FACE sequence of 1 thru 8. Thus, including only including 'FrontDoor_3Dpas' set, from those four, in order to converse space

---

A convenient feature of 'shapefile' attribute data is that it gets stored in a .DBF sidecar file, based on some old version of dBase, making the data easy to view for spreadsheet-type apps which directly support reading of DBF files (Excel, for example).

---

Old dBASE does have a date-type, "D", which is simply the 8-byte ASCII YYYYMMDD. Which is basically what Walls is exporting, though I'm not clear if it's designated as a 'date' field. (New versions of dBASE, irrelevant to shapefiles, use a 32-bit date & 32-bit time.)

Binary date formats have the downside of potentially creating additional hoops depending on the downstream consumer. YYYYMMDD and Compass YYYY, MM and DD typically require but one hoop to convert to internal date-time binary format in downstream consumers.

---

Based on the eyeballing the data in the attached sample, I'd agree that COMPASS_NMI, NMA, EMI, EMA, VMI and VMA represent the bounding-box coordinate.

[Joe Walko]

zip file with sample walls and compass exported shapefiles