[Antelope] Steim Compression Error

[Daniel Quinlan]

Hi Hank,

The maximum difference allowed in steim 2 compression is 2^30, and  
possibly the value of your
first sample in this particular waveform, multiplied by your calib, is  
greater than that.  The first
difference saved in the compressed data is the difference between the  
first sample and zero.  I suspect
this is the reason you are getting an error message.

Steim compression is intended to save integer data from a data logger,  
not compress arbitrary data,
and especially not floating point data.  So multiplying by calib and  
then saving the data in integer
format is a bit questionable.  You might instead save the calib  
corrected data as floating point.

-- danq 

On Nov 17, 2008, at 10:14 AM, Hank Ratzesberger wrote:

>
> Hi Kent,
>
> We have finally created the smallest example the re-creates the
> compression error.  A zip file is available at:
>
> http://nees.ucsb.edu/pdf/problem_Example.zip
>
> with instructions in the readme for the checks we did
> on the file and the results.
>
> We note that the saved file has one less sample, and the
> first sample is bad.  Otherwise, the program is not much
> more than the following:
>
> tr_loadcss()
> tr_apply_calib()
> tr_savewf()
>
> Regards,
> Hank
>
>
> On Nov 10, 2008, at 2:31 PM, Hank Ratzesberger wrote:
>
>> Thanks for your interest Kent.
>>
>> This is the program we use to apply the calib and
>> copy the data to the new database.  (We use a very
>> similar program which also calls rotate_to_standard.)
>> It is called from a Perl script.
>>
>> Probably a good sign, but I cannot reproduce it in a
>> minimal example.  We call this program repeatedly from
>> a script that identifies all the true ZNE channels to
>> apply the calib to the actual data.
>>
>> So, I think the issue is not the library, but using it
>> under load of some kind.  Let me get back to you.
>>
>> Thanks,
>> Hank
>>
>>
>>
>> <trcopy.c>
>>
>> Attached is the original file.
>>
>> <WLA_HLE_00_14179736.msd>
>>
>>
>> Below is a trace showing the out of place first sample:
>>
>>
>> <Picture 1.png>
>>
>>
>>
>> On Nov 8, 2008, at 4:26 PM, Dr. Kent Lindquist wrote:
>>
>>>
>>> Hello Hank,
>>>
>>> Not to imply that I'm promising to debug and fix this, but the  
>>> next step that would be nice is to make a very small, completely  
>>> self-contained data-set; make a tiny little program that runs on  
>>> it to reproduce the error; then explain precisely how the output  
>>> of the script differs from what you expect. Also exactly where and  
>>> from what would you 'drop the first data-point'? From the input,  
>>> the processed trace structure, or the output? The reproducible  
>>> example might help someone in the community see exactly what's  
>>> occurring and help you resolve resolve it.
>>>
>>> Best regards,
>>> Kent
>>>
>>>
>>>
>>> On Nov 8, 2008, at 6:07 AM, Hank Ratzesberger wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> We have a program that calls the library routine trapply_calib  
>>>> then saves
>>>> the data to a new database.
>>>>
>>>> We recently noticed that some resulting traces have point  
>>>> "artifacts"
>>>> that are way out of range and looked further to find this message
>>>> in our log files:
>>>>
>>>> trcopy_osx: The difference -2147483648 between 0 and -2147483648  
>>>> near sample #0 cannot be represented in a Steim level 2  
>>>> compressed record
>>>>
>>>> The program is running on an Intel Mac running Leopard.
>>>> We are running some more tests to see if the problem does not
>>>> occur on other platforms.
>>>>
>>>> We do not fill gaps.  The problem occurs on files that have no  
>>>> gaps.
>>>>
>>>> Any thoughts on why we get this error and what we might do about  
>>>> it?
>>>> We could always drop the first data point -- if we knew how.
>>>>
>>>> Thanks,
>>>> Hank
>>>>
>>>>
>>>> Hank Ratzesberger
>>>> NEES at UCSB
>>>> Institute for Crustal Studies,
>>>> University of California, Santa Barbara
>>>> 805-893-8042
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Antelope mailing list
>>>> Antelope at brtt.net
>>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>>
>>> --
>>> Dr. Kent Lindquist                     kent at lindquistconsulting.com
>>> Lindquist Consulting, Inc.
>>> 59 College Rd. Suite #7
>>> Fairbanks, AK 99701              Phone/FAX 907-457-2374
>>>
>>> http://www.lindquistconsulting.com
>>>
>>>
>>> _______________________________________________
>>> Antelope mailing list
>>> Antelope at brtt.net
>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>
>> Hank Ratzesberger
>> NEES at UCSB
>> Institute for Crustal Studies,
>> University of California, Santa Barbara
>> 805-893-8042
>>
>>
>>
>>
>>
>> _______________________________________________
>> Antelope mailing list
>> Antelope at brtt.net
>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>
> Hank Ratzesberger
> NEES at UCSB
> Institute for Crustal Studies,
> University of California, Santa Barbara
> 805-893-8042
>
>
>
>
>
>
> _______________________________________________
> Antelope mailing list
> Antelope at brtt.net
> http://brtt.net/mailman/listinfo/antelope_brtt.net