[Antelope] DBBUILD help
Jennifer Eakins
jeakins at ucsd.edu
Wed Feb 11 12:08:10 CST 2009
Hello Ali,
With a minor modification where I replaced the name of the Trillium 40
response file (from Trillium-40-NEW to trillium_40), I cat'd your
attached *.DBBUILD files together and successfully ran dbbuild:
% cat *D > my-dbbuild
% dbbuild -b test my-dbbuild
% dbverify test
0 problems
I can't tell you if the information in your files is correct, but the
dbbuild batch file setup seems ok. If you have a response file for
"Trillium-40-NEW" it needs to reside in your distribution under:
$ANTELOPE/data/instruments/sensors/ and with the name Trillium-40-
NEW.pf. Please note that the current version of Antelope is 4.10 and
that there was a recent correction to the default trillium_40
response. There have also been numerous updates to the dbbuild
program since the 4.8 release.
I am uncertain what your next concern/question is. You did not attach
the dbbuild files for the 5 stations from the AE network (I assume
AJN, ALN, MSF, SHM, and UMQ). Presumably you want all of the stations
in your operational dbmaster (DN_*, AE_*, II_*, etc.). If you need to
maintain a separate "dn" database for only stations that have the "DN"
network code, I would strongly suggest changing your setup (see
below). Alternatively, the 4.10 version of Antelope has an updated
mk_dataless_seed program that allows you to subset a database to
generate a single station dataless SEED volume.
Below is a portion of a write-up that I put together based on
operations for the USArray TA where metadata is gathered from multiple
networks as well as maintained for all TA stations past and present
(currently 632). It details a way to generate a single dbmaster from
separate regional networks using both dbbuild and dataless SEED
volumes. It involves a different setup than you have, but perhaps it
will answer your concerns. Once you understand the "Directory
structure", you can probably skip everything until the section "How to
merge all network dbmasters" unless you want to see some specifics
about screen out stations from a dataless SEED volume provided by
another regional/global network operator. Note that due to the
complexity of the network that I deal with, I have made no attempt to
put all of the commands into a Makefile.
--Jennifer Eakins
=====================================
Directory structure
I have found that for complex and dynamic databases such as the TA,
you need to have a “working” area separate from rtsystems/project/
dbmaster/. This allows you to take your time in making changes to the
TA database, as well as keep track of changes to the regional networks
in an area that is (hopefully) less dynamic than the TA stations.
Separating out the TA stations, also allows for generation of a
network specific dataless SEED volume that does not include stations
that are the responsibility of another.
I recommend that the person in charge of metadata updates and
collection create a staging area:
/your/path/rtsystems/project/pre-dbmaster/
Within the pre-dbmaster directory you will find separate directories
for each regional network, global network, and the TA stations. For
example:
% cd pre-dbmaster/
% pwd
/your/path/rtsystems/project/pre-dbmaster
% ls
all_merge/ CONTRIB_NETWORKS_merge/
az_dbbuild/ iu_only/
bk_only/ ta_dbbuild/
ci_only/
The following is a brief explanation of what each directory contains.
Additional details will be provided in the “Regional networks” and “TA
network” descriptions section of this document.
all_merge - contains the final combined regional, backbone, and TA
dbmasters. The db in this directory will eventually replace the
dbmaster in /your/path/rtsystems/project/dbmaster/
CONTRIB_NETWORKS_merge - contains the combined regional and global
dbmasters. The db in this directory will eventually be combined with
the database in ta_dbbuild to form the dbmaster in all_merge.
az_dbbuild - contains a dbbuild file and the dbmaster for the three
Anza stations, MONP, MONP2, and PFO.
bk_only - contains a single dataless SEED volume for the entire BK
(BDSN) network, a pf for sorting what stations are contributing, as
well as the dbmaster created for the stations that contribute to the TA.
ci_only - contains a directory with single station dataless SEED
volumes for the CI network (SCSN), a pf for sorting what stations are
contributing, as well as the dbmaster created from these individual
dataless volumes.
iu_only - contains a dataless SEED volumes for the IU network (GSN
maintained by ASL), a pf for sorting what stations are contributing,
as well as the dbmaster created for the station that contributes to
the TA.
ta_dbbuild - contains multiple directories including backups of
previous builds, a directory containing “individual” dbbuild files, a
dataless directory with past dataless SEED volumes, and the dbmaster
created from the individual dbbuild databases.
Regional networks
Each regional network has it’s own way of distributing metadata,
mostly in the form of a dataless SEED. Some are available via ftp,
others only by http download. They can be distributed as a single
dataless for the entire network, or individual station dataless. I
will try to describe where I obtain the dataless for each network, and
how I build the dbmaster for that individual network.
AZ (Anza) stations: MONP, MONP2, PFO
UCSD (Vernon group) operates the Anza network (network code AZ) and is
responsible for the metadata. As of December 2007, we are not
maintaining single station dataless, or even an updated whole network
dataless, for the Anza network. So, in order to get the proper
metadata for these contributing stations, I created a dbbuild file
that would cover the stations of interest. Note that this database
does not have the proper install times for PFO and MONP, but cover
only the epochs when the stations were in use for the TA (i.e. after
3/01/2004). Eventually, the plan is for the entire Anza network to
switch to using single station dataless SEED volumes (hopefully
2009). At that point, the procedures below will not be necessary, and
there will be a “pick up” point described for obtaining the dataless
SEED volumes.
No updates should be necessary here, but if there is a sensor or
datalogger swap at one of these sites, follow these procedures:
Make a backup directory and copy previous dbmaster and dbbuild files
to it:
% pwd
/your/path/rtsystems/project/pre-dbmaster/az_dbbuild
% mkdir 2009-042
% mv az_monp_pfo* response 2009-042
Edit the my-dbbuild file and add in the changes
% vi my-dbbuild
Run dbbuild to generate newly updated database:
% dbbuild –b az_monp_pfo my-dbbuild
You should now have a database, az_monp_pfo, which is ready for
merging with the other regional networks.
BK (BDSN) stations: BDM, CMB, CVS, FARB, GASB, HOPS, HUMO, JCC, JRSC,
KCC, MCCM, MNRC, MOD, ORV, PACP, PKD, POTR, WDC, WENL, YBH
UCB operates the BDSN network (network code BK) and is responsible for
the metadata. As of December 2007, the BK network is no longer
contributing to the TA. The stations listed above include all
stations that ever contributed to the TA. Some stations contributed
for the entire time range that BK contributed, others only for a short
time. Regardless of how long they contributed, metadata for all of
the above stations has to be included.
Berkeley produces a network-based dataless SEED volume. It is
available via http pickup from:
http://www.ncedc.org/ftp/pub/doc/BK.info/BK.dataless.seed
or from ftp pickup via:
% ftp www.ncedc.org
% cd pub/doc/BK.info
% get BK.dataless.seed
As this network is no longer contributing it should be “static” and no
updates needed. However, here are the procedures if some future
update is needed.
Make a backup directory and copy previous dbmaster and dataless files
to it:
% pwd
/your/path/rtsystems/project/pre-dbmaster/bk_only
% mkdir 2009-042
% mv bk_tmp* BK.dataless.seed response 2009-042
Collect the updated dataless SEED volume via one of the above methods:
% ftp www.ncedc.org
Run seed2db:
% seed2db –respdir response –stagedir \
response/stage_BK –chansift netstachanlocs.pf \
BK.dataless.seed bk_tmp
Note that I use a network specific stage directory. This helps in
later merging of the databases. Also see that we use a
netstachanlocs.pf file. This file has a list of accepted
net_sta_chans that will be selected from the dataless SEED for
conversion into a css3.0 db: it should not need to be modified for a
“closed” network like this one. The netstachanlocs.pf file looks like:
accept &Tbl{
BK_BDM_[BL]H.*
BK_BDM_L[KD]S.*
BK_BDM_UEP.*
BK_BDM_UKI.*
BK_BDM_UM.*
BK_BDM_LOG.*
BK_BDM_ACE.*
BK_CMB_[BL]H.*
BK_CMB_L[KD]S.*
BK_CMB_UEP.*
BK_CMB_UKI.*
BK_CMB_UM.*
BK_CMB_LOG.*
BK_CMB_ACE.*
...
}
There are multitudes of complaints about the FIR delay for these
dataless. I have not tracked them down. Here is an example of the
error:
2007-363 16:00:22 seed2db *complain*: SEEDERROR - Improbable value for
FIR delay.
2007-363 16:00:22 seed2db *complain*: Resetting stage 3 FIR delay
from 0.00(0.000000) to 2.27(0.000071) for BK_BDM_LHE
1999:337:19:26:00.000 -> 2000:066:00:01:00.000.
2007-363 16:00:22 seed2db *complain*: SEEDERROR - Asymmetric FIR
coefficients in wrong order.
2007-363 16:00:22 seed2db *complain*: Reversing order of stage 4
coefficients for BK_BDM_LHE 1999:337:19:26:00.000 ->
2000:066:00:01:00.000.
2007-363 16:00:22 seed2db *complain*: SEEDERROR - Asymmetric FIR
coefficients in wrong order.
2007-363 16:00:22 seed2db *complain*: Reversing order of stage 5
coefficients for BK_BDM_LHE 1999:337:19:26:00.000 ->
2000:066:00:01:00.000.
You should now have a database, bk_tmp, which is ready for merging
with the other regional networks.
CI (SCSN) stations: ADO, ARV, BBR, BC3, BCC, BEL, BFS, CIA, CWC, DAN,
DEC, DNR, DVT, EDW, EDW2, FMP, FUR, GLA, GMR, GOR, GRA, GSC, HEC, IRM,
ISA, LUG, LRL, MLAC, MPI, MPM, MPP, MUR, NEE, NEE2, OSI, PDM, RCT,
RRX, SBC, SCI2, SCZ2, SDP, SDR, SHO, SRMM, SNCC, SWS, TIN, TUQ, VES, WER
Caltech operates the SCSN network (network code CI) and is responsible
for the metadata. As of December 2007, CI is still a contributor to
the TA. The stations listed above include all stations that ever
contributed to the TA. Some stations have contributed for the entire
time range of the deployment (i.e. BBR), others only for a short time
(ADO), and some stations were closed and replaced (i.e. NEE2).
Regardless of how long they contributed, metadata for all of the above
stations has to be included. Early on in 2004, some stations event
contributed 100 sps data rather than the standard 40 and 1sps.
Because of this, all metadata must be kept.
Caltech produces a single station dataless SEED volume. They are
available via http pickup from:
http://www.data.scec.org/ftp/stations/seed/
or from ftp pickup via:
% ftp scec.gps.caltech
% cd pub/stations/seed
% get dataless.CI.BBR
At some point, I started writing a script that would check for remote
dataless files that had been updated more recently than those I had
locally, but I never got it to work consistently. Thus, it is left as
a (rather tedious) human process to see if there are any newly updated
dataless that need to be retrieved.
Assuming that there were updates, here are the procedures:
Make a backup directory and copy previous dbmaster and dataless files
to it:
% pwd
/your/path/rtsystems/project/pre-dbmaster/ci_only
% mkdir 2009-042
% mv ci_tmp* response 2009-042
** Note that I do not save a backup of any dataless.
Collect the updated dataless SEED volume via one of the above methods:
% cd CI_dataless
% ftp scec.gps.caltech
Run seed2db within a foreach loop:
% cd ../
% foreach file (CI_dataless/*dataless*)
echo $file
seed2db –respdir response –stagedir \
response/stage_CI –chansift chansift.pf \
$file ci_tmp
end
Note that I use a network specific stage directory. This helps in
later merging of the databases. Also see that we use a chansift.pf
file. This file has a list of accepted net_sta_chans that will be
selected from the dataless SEED for conversion into a css3.0 db: it
might need to be modified if a station is closed and replaced with a
new one. The only likely change would be the addition of a new
station: if a station is closed/removed, we track its removal from
the TA by modifying the deployment table (more on that later).
There are multitudes of complaints about the FIR delay for these
dataless. I have not tracked them down. Here is an example of the
error:
2007-363 15:56:44 seed2db *complain*: SEEDERROR - Improbable value for
FIR delay.
2007-363 15:56:44 seed2db *complain*: Resetting stage 3 FIR delay
from 319.50(0.015975) to 0.00(0.000000) for CI_BBR_BHE
2000:166:22:00:00.000 -> 2004:085:04:39:00.000.
2007-363 15:56:44 seed2db *complain*: SEEDERROR - Improbable value for
FIR delay.
2007-363 15:56:44 seed2db *complain*: Resetting stage 4 FIR delay
from 79.50(0.079500) to 0.00(0.000000) for CI_BBR_BHE
2000:166:22:00:00.000 -> 2004:085:04:39:00.000.
You should now have a database, ci_tmp, which is ready for merging
with the other regional networks.
IU (ASL GSN) stations: TUC
ASL (Arizona Seismic Lab) operates the IU portion of the GSN network
(network code IU) and is responsible for the metadata. As of December
2007, IU is a contributor of a single station to the TA. I am not
aware of any mechanism (email or otherwise) that informs downstream
users of any possible changes to the dataless. Changes are
infrequent, so perhaps checking quarterly for any updates is adequate.
ASL produces single station dataless SEED volumes. They are only
available via ftp
% ftp aslftp.cr.usgs.gov
% cd pub/dataless
% get IU.dataless
Assuming that there was an update to TUC (you can look at the listing
for the individual dataless to see if there was an update), here are
the procedures:
Make a backup directory and copy previous dbmaster and dataless files
to it:
% pwd
/your/path/rtsystems/project/pre-dbmaster/iu_only
% mkdir 2009-042
% mv iu_tmp* response 2009-042
** Note that I do not save a backup of any dataless.
Collect the updated dataless SEED volume via one of the above methods:
% ftp aslftp.cr.usgs.gov
Run seed2db:
% seed2db –respdir response –stagedir \
response/stage_IU –chansift chansift.pf \
IU.dataless iu_tmp
Note that I use a network specific stage directory. This helps in
later merging of the databases. The chansift.pf file is necessary as
this network dataless has additional stations/channels that do not
contribute to the TA, and we do not need them in our database. Errors
look like:
2007-363 15:49:45 seed2db *complain*: SEEDERROR - Product of stage
gains (3692614771.6232) does not equal final sensitivity
(3641000000.0000) for IU_TUC_LHE_00 1998:309:19:00:00.000 -> NULL.
There seems to be a problem if you attempt to use the individual
dataless (i.e. DATALESS.IU_TUC.seed). The errors from using the
single station dataless look like:
2007-363 15:53:49 seed2db *notify*: seed2db : $Revision: 1.21 $ $Date:
2007/02/14 18:12:02 $
2007-363 15:53:49 seed2db *log*: scan_int: Failed integer input: '-r--
r-'
2007-363 15:53:49 seed2db *log*: sdblk_parseblk:
scan_int(logical_sequence,'-r--r-') error at 0 bytes offset
2007-363 15:53:49 seed2db *fatal*: SEEDERROR - sdblk_parseblk
(DATALESS.IU_TUC.seed) error.
You should now have a database, iu_tmp, which is ready for merging
with the other regional networks.
TA network
This is by far the most complicated portion of the usarray dbmaster.
The complications result from the shear number of stations,
potentially up to 400 TA installed at any time, over 2000 stations by
the end of the project, and the dynamic nature of the network with
stations operating for two years or less and both installations and
removals occurring constantly. However, we are lucky in that there
are only two datalogger configuration and three different sensor types.
The procedure for updating the TA dbbuild file has undergone multiple
revisions as the network has grown. Initially, I used a single
dbbuild file that contained information in chronologic order for all
stations. This grew hard to manage and I moved to a single file
sorted alphabetically and then by time. This also proved unwieldy and
aggravating (i.e. if any mistakes were made during an update for a
single station, the entire dbbuild run had to be redone). I briefly
used a technique where I ran dbbuild only on a “dbbuild-snippet” that
contained only the updated information. This was a speedy way of
updating, but some updates tended to get lost or not propagated to the
main dbbuild file and I found it was not workable for some revisions
needed for earlier epoch times.
The current method I employ is to maintain ~30 dbbuild files
containing all stations which match a single letter/number. For
instance, A04A, A05A, etc. through A21A are found in the AXXX-dbbuild
file. Each station grouping has it’s own database generated by
running dbbuild, and then all databases are grouped together into a
ranged database and finally into a combined TA-only database. I am
not convinced this is the best way to do this (in terms of time/
efficiency), but it seems to avoid some aggravating mistakes.
If you look at the current build area, you will see backup directories
(i.e. 2008-357/), a database called ANF_TA-only, a directory called
dataless/, a directory called individual_dbbuild/, some scripts
(build_all and merge_grp_dbs), and a few other miscellaneous files and
directories. The short summary is that the ANF_TA-only is the “final”
TA station only database from which I build a dataless SEED volume to
pass to downstream users. The dbbuild files are found in the
individual_dbbuild directory with the results from each individual
dbbuild run found in individual_dbbuild/$letter/. The output dataless
SEED volume is gzip’d and saved in the dataless directory.
<... skipping over many gory details on building the ANF_TA-only
database...>
How to merge all network dbmasters
Now that you have all of the contributing networks and the TA network
databases up-to-date, you can merge them in preparation of replacing
the main dbmaster on anfops. Here are the steps:
Prep for merge of all contributing networks:
% pwd
/your/path/rtsystems/project/pre-dbmaster/
% cd CONTRIB_NETWORKS_merge/
% mkdir 2009-042
% mv usarray.* response/* 2009-042
Merge all contributing non-TA networks:
% dbmerge ../az_dbbuild/az_monp_pfo usarray
% dbmerge ../bk_only/bk_tmp usarray
% dbmerge ../ci_only/ci_tmp usarray
% dbmerge ../iu_only/iu_tmp usarray
You now have a rebuilt database in CONTRIB_NETWORKS_merge that
contains all contributing networks. In theory, these stations do not
get updated often so it is convenient to keep it separate from the
“all_merge” db.
Prep for all_merge:
% pwd
/your/path/rtsystems/project/pre-dbmaster/
% cd all _merge/
% mkdir 2009-042
% mv usarray.* response/* 2009-042
% cp ../CONTRIB_NETWORKS_merge/usarray.* .
% cp –R ../CONTRIB_NETWORKS_merge/response .
Merge in TA data:
% dbmerge ../ta_only/ANF_TA-only usarray
We’re almost done! Now we need to stop various real-time systems and
then update the main dbmaster.
<... procedure below is different for ANF/TA. Simplified instructions
follow...>
Stop any processes that access the dbmaster directory. This includes
any analyst processing via dbpick or dbloc2, orbassoc, and possibly
orbdetect
% pwd
/your/path/rtsystems/project/
% orb2db_msg pause
% vi rtexec.pf
This orb2db_msg command pauses the orb2db and orb2dbt processes for
the realtime systems who's rtexec.pf resides in the pwd. When you
edit rtexec.pf, turn off the orbdetect and orbassoc processes (change
"yes" to "no" or "1" to "0").
Now that the real-time systems are no longer running processes that
look at the dbmaster, it can be updated.
% pwd
/your/path/rtsystems/project/dbmaster/
% mkdir 2009-042
% cp usarray.* 2009-042
% mv response/* 2009-042
% cp ~/rtsystems/pre-dbmaster/all_merge/usarray.* .
% cp –R ~/rtsystems/pre-dbmaster/all_merge/response .
You will have to say “yes” to overwrite the usarray.* files. It is
also somewhat redundant to make a backup of the dbmaster files here if
you have already made one in the all_merge area.
Make sure the orb2db and orb2dbt processes are allowed to continue.
% pwd
/your/path/rtsystems/project/
% orb2db_msg pause
Remember to edit your rtexec.pf to turn your orbdetect and orbassoc
process back on.
On Feb 10, 2009, at 12:25 AM, Ali Shaaban Megahed wrote:
> Hi All
>
> I am using Antelope 4.8 for real time data acquisition, analysis and
> share data with neigbooring networks. It is my first time to update
> the database with new stations, the database consists of 4 stations
> ASU, HAT, NAZ, FAQ which belong to DN network as shown in the
> attached Makefile and the other stations as shown in other.pf file.
> we are trying to share data with AE network which is consists of 5
> stations (attached the sta.DBBUILD files) which I need to add to the
> database, I do not have the dataless seed file for these stations so
> I prepared the STA-DBBUILD files for the 5 stations of AE network, I
> just add the information to the STA1.DBBUILD and rename the files by
> the stations name as attached, could you please check them…
> The Makefile attached after I modify it with the AE network
> stations…….the problem is that the new stations are not belong to DN
> network to add their names at the end of makefile….
>
> Could you please send me your recommendation after you check the
> attached files…
>
>
>
>
> Best wishes
>
> Ali
>
> Seismologist
> Dubai Seismi Network
> Survey Dapartment
> Dubai Municipality, Dubai, UAE
> asmegahed at dm.gov.ae
>
> Our Vision
> To create an excellent city that provides the essence of success and
> comfort of living.
>
> Disclaimer:
> This Electronic Mail and any files transmitted with it are
> confidential
> and intended solely for the use of the individual or entity to whom
> they
> are addressed. If you are not an addressee, or have received the
> message
> by error, please notify the sender via E-Mail or over the
> telephone and
> delete this e-mail. You are not authorized to read, copy,
> disseminate,
> distribute or use this E-Mail or any of its attachment in any
> way. Any
> views or opinions presented in this email are solely those of the
> author
> and do not necessarily represent Dubai Municipality. The recipient
> should
> check this email and any attachments for the presence of viruses/
> worms.
> Dubai Municipality accepts no liability for any damage caused
> by any
> virus/worms transmitted by this email.
>
> Dubai Municipality, Dubai, UAE, http://www.dm.gov.ae/
>
> <AJN-DBBUILD><ALN-DBBUILD><MSF-DBBUILD><UMQ_DBBUILD><SHM-
> DBBUILD
> ><Makefile><other.pf>_______________________________________________
> Antelope mailing list
> Antelope at brtt.net
> http://brtt.net/mailman/listinfo/antelope_brtt.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://brtt.net/pipermail/antelope_brtt.net/attachments/20090211/4e09b7a4/attachment-0001.html>
More information about the Antelope
mailing list