[Antelope] rtdbclean advisory

[Glenn Biasi]

For rtdbclean users in real-time acquisition:

We have encountered a condition using rtdbclean that removes more data 
than is intended.  The context is an acquistion directory that takes in 
about 2 Gb/day.  If the rtdbclean "min_disk_free" minimum is not 
satisfied, the model is to delete a day of waveforms, then test whether 
min_disk_free is now less than the minimum required free space.  On 
Solaris, however, "rm" and its allies are not blocking calls.  That is, 
one might issue a command to remove 10 Gb, get the prompt back 
immediately, and find little immediate change in the free space.  Inside 
the script, when the "df" call next tests for free space, the return 
value is only the original free space plus what has been freed since the 
rm command.  This sets up a race condition between the data removal and 
the script test-and-remove cycle.  In our case we were down to ~600 Mb 
with a min_free_space of 1000 Mb.  rtdbclean issued "rm" orders for six 
or seven day's of data before the disk cleared above 1 Gb.  When the 
accumulated "rm" commands concluded, we had ~14 Gb free.  It has 
happened more than once, and has removed some data we cared about.  I 
have also tested the Solaris behavior with simpler scripting examples. 
Note that if, say, the free space is only a little under min_disk_free 
(or maybe one has really fast disks), rtdbclean may exhibit none of the 
above-described symptoms and perform as expected.

To fix it I added a hard-coded a 1 minute per Gb sleep before testing 
the space again.  Around line 105, under:

$tmbytes += $nbytes = &clean_disk() ;

I added the estimator:

$time2act = $tmbytes / 17 ;   # one minute per gigabyte

and a sleep:

sleep $time2act ;

This seems to yield the desired effect.  I would defer to others on 
whether a more elegant solution is worth the trouble.  I have no idea 
how or whether the above race condition also arises in Linux or Mac 
implementations.


Glenn


-- 
Glenn Biasi
University of Nevada Reno
Seismological Laboratory,  MS-174
Reno, NV 89557
ph: (775) 784-4576  fax: (775) 784-4165