[Antelope] Steim Compression Error
Hank Ratzesberger
hankr at crustal.ucsb.edu
Tue Nov 25 11:50:49 CST 2008
Thanks Richard and everyone.
Changing the compression to STEIM1 in the trdefaults.pf file
does eliminate the error message and out of range first sample.
We do "loose" a sample, however, the converted file is one
sample shorter. It doesn't affect anything else we do, so its
one for the errata.
Cheers,
Hank
On Nov 20, 2008, at 11:09 AM, Richard Baldwin wrote:
> Hi all,
>
> A very minor point: the maximum difference allowed in Steim 2
> compression is 30 bits long. The actual range is -2^29 to +2^29-1,
> as stated in the SEED V2.4 (2007-10) manual, page 136.
>
> Richard
>
> Daniel Quinlan wrote:
>> Hi Hank,
>>
>> The maximum difference allowed in steim 2 compression is 2^30, and
>> possibly the value of your
>> first sample in this particular waveform, multiplied by your
>> calib, is greater than that. The first
>> difference saved in the compressed data is the difference between
>> the first sample and zero. I suspect
>> this is the reason you are getting an error message.
>>
>> Steim compression is intended to save integer data from a data
>> logger, not compress arbitrary data,
>> and especially not floating point data. So multiplying by calib
>> and then saving the data in integer
>> format is a bit questionable. You might instead save the calib
>> corrected data as floating point.
>>
>> -- danq
>> On Nov 17, 2008, at 10:14 AM, Hank Ratzesberger wrote:
>>
>>>
>>> Hi Kent,
>>>
>>> We have finally created the smallest example the re-creates the
>>> compression error. A zip file is available at:
>>>
>>> http://nees.ucsb.edu/pdf/problem_Example.zip
>>>
>>> with instructions in the readme for the checks we did
>>> on the file and the results.
>>>
>>> We note that the saved file has one less sample, and the
>>> first sample is bad. Otherwise, the program is not much
>>> more than the following:
>>>
>>> tr_loadcss()
>>> tr_apply_calib()
>>> tr_savewf()
>>>
>>> Regards,
>>> Hank
>>>
>>>
>>> On Nov 10, 2008, at 2:31 PM, Hank Ratzesberger wrote:
>>>
>>>> Thanks for your interest Kent.
>>>>
>>>> This is the program we use to apply the calib and
>>>> copy the data to the new database. (We use a very
>>>> similar program which also calls rotate_to_standard.)
>>>> It is called from a Perl script.
>>>>
>>>> Probably a good sign, but I cannot reproduce it in a
>>>> minimal example. We call this program repeatedly from
>>>> a script that identifies all the true ZNE channels to
>>>> apply the calib to the actual data.
>>>>
>>>> So, I think the issue is not the library, but using it
>>>> under load of some kind. Let me get back to you.
>>>>
>>>> Thanks,
>>>> Hank
>>>>
>>>>
>>>>
>>>> <trcopy.c>
>>>>
>>>> Attached is the original file.
>>>>
>>>> <WLA_HLE_00_14179736.msd>
>>>>
>>>>
>>>> Below is a trace showing the out of place first sample:
>>>>
>>>>
>>>> <Picture 1.png>
>>>>
>>>>
>>>>
>>>> On Nov 8, 2008, at 4:26 PM, Dr. Kent Lindquist wrote:
>>>>
>>>>>
>>>>> Hello Hank,
>>>>>
>>>>> Not to imply that I'm promising to debug and fix this, but the
>>>>> next step that would be nice is to make a very small,
>>>>> completely self-contained data-set; make a tiny little program
>>>>> that runs on it to reproduce the error; then explain precisely
>>>>> how the output of the script differs from what you expect. Also
>>>>> exactly where and from what would you 'drop the first data-
>>>>> point'? From the input, the processed trace structure, or the
>>>>> output? The reproducible example might help someone in the
>>>>> community see exactly what's occurring and help you resolve
>>>>> resolve it.
>>>>>
>>>>> Best regards,
>>>>> Kent
>>>>>
>>>>>
>>>>>
>>>>> On Nov 8, 2008, at 6:07 AM, Hank Ratzesberger wrote:
>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We have a program that calls the library routine trapply_calib
>>>>>> then saves
>>>>>> the data to a new database.
>>>>>>
>>>>>> We recently noticed that some resulting traces have point
>>>>>> "artifacts"
>>>>>> that are way out of range and looked further to find this message
>>>>>> in our log files:
>>>>>>
>>>>>> trcopy_osx: The difference -2147483648 between 0 and
>>>>>> -2147483648 near sample #0 cannot be represented in a Steim
>>>>>> level 2 compressed record
>>>>>>
>>>>>> The program is running on an Intel Mac running Leopard.
>>>>>> We are running some more tests to see if the problem does not
>>>>>> occur on other platforms.
>>>>>>
>>>>>> We do not fill gaps. The problem occurs on files that have no
>>>>>> gaps.
>>>>>>
>>>>>> Any thoughts on why we get this error and what we might do
>>>>>> about it?
>>>>>> We could always drop the first data point -- if we knew how.
>>>>>>
>>>>>> Thanks,
>>>>>> Hank
>>>>>>
>>>>>>
>>>>>> Hank Ratzesberger
>>>>>> NEES at UCSB
>>>>>> Institute for Crustal Studies,
>>>>>> University of California, Santa Barbara
>>>>>> 805-893-8042
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Antelope mailing list
>>>>>> Antelope at brtt.net
>>>>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>>>>
>>>>> --
>>>>> Dr. Kent Lindquist
>>>>> kent at lindquistconsulting.com
>>>>> Lindquist Consulting, Inc.
>>>>> 59 College Rd. Suite #7
>>>>> Fairbanks, AK 99701 Phone/FAX 907-457-2374
>>>>>
>>>>> http://www.lindquistconsulting.com
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Antelope mailing list
>>>>> Antelope at brtt.net
>>>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>>>
>>>> Hank Ratzesberger
>>>> NEES at UCSB
>>>> Institute for Crustal Studies,
>>>> University of California, Santa Barbara
>>>> 805-893-8042
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Antelope mailing list
>>>> Antelope at brtt.net
>>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>>
>>> Hank Ratzesberger
>>> NEES at UCSB
>>> Institute for Crustal Studies,
>>> University of California, Santa Barbara
>>> 805-893-8042
>>>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Antelope mailing list
>>> Antelope at brtt.net
>>> http://brtt.net/mailman/listinfo/antelope_brtt.net
>>
>>
>> _______________________________________________
>> Antelope mailing list
>> Antelope at brtt.net
>> http://brtt.net/mailman/listinfo/antelope_brtt.net
Hank Ratzesberger
NEES at UCSB
Institute for Crustal Studies,
University of California, Santa Barbara
805-893-8042
More information about the Antelope
mailing list