Importing Flags from other software

Lots of people use a particular software package to process their data, but then want to do some specialized processing in our software. It's easy enough to import the raw data into our software, but most people don't want to "re-clean" the data. This set of instructions can guide you through how to "export the cleaning" from a software package such as HIPS, and then to apply the cleaning the data in our format.
  1. Export a list of soundings that have been rejected using their generic export tool. Make sure you specify "rejected" and not "accepted". The only fields you need to export are profile (ping) and beam number. This needs to be done separately for every file (or put the survey line name in it as well and then use some awk/sed/grep wizardry to separate them after the fact). Your text files should look like this:
    1 2
    1 3
    1 11 
    
    The above example indicates that beams 2,3 and 11 have been flagged for ping #1.

  2. Convert the text file into binary for application to OMG .merged files. You do this with make_flags:
    	make_flags -rejected -flag_file flags_file.txt -in file.merged
    
    Note that "flags_file.txt" needs to be the appropriate flag file for "file.merged". This will create a file.flag_bk to go along with file.merged, this gets applied to your soundings in the next step.

  3. Apply the flags to the .merged file. The traditional route at this stage would be to use dumpFlags, but this barely ever works correctly. In fact, this ONLY works correctly if ping number 1 in your processing software corresponds to ping number 1 in OMG. This is hardly ever the case, for example HIPS drops out pings that are un-navigable (first few pings before the first nav fix in the raw data file)...but we keep them in OMG format. So, ping #1 in HIPS hardly ever corresponds to ping #1 in OMG.

    The solution is to "guess" the offset between your software's ping numbering and our ping numbering. As the offset can vary between survey lines, I modified swathed to help you guess the offset.

    	swathed file.merged
    
    Unflag any previous cleaning with a capital U key, and then hit F10. This will attempt to load the .flag_bk file and will automagically guess the offset. What you'll see is your dirty data being cleaned for you. This works well with Reson data since they preserve all range measurements, even the bad outliers (Simrad filters the bad outliers usually).

    If you think it applied the appropriate flags, you just space bar your way through the remainder of the file. If you think you have to nudge the offset, you can use F9 and F11 to decrement and increment the offset, the offset will be reported on the command line. You can reset the offset to zero by hitting F12. Once you've started this process, it will automatically apply the flags as you space bar through the file. The tweaking of the auto-guessed offset may to be repeated for every file.

    In summary, here are the magic keystrokes:

I fully intend to automate the above procedure, but the automagic offset guessing algorithm requires some fine-tuning before I trust it in automation, particulary for relatively clean datasets.


Last modified by: J. Beaudoin, Oct. 6th, 2008