Installing the CREW software package

From CrewWiki

(Difference between revisions)

Latest revision as of 16:54, 27 March 2014

Several steps have to be completed before you can use the CREW software package if you like to intercompare SEVIRI retrievals:

Installation procedure for SEVIRI dataset inter-comparison

Registration at ICARE

Please send us an email to register at the ICARE service.

Download the CREW software

Go to a folder, where you would like to install the CREW software (4.4MB).
Download an unpack the CREW software package crewsoftware2014-03.tar.bz2
unpack the software archive (for Linux use tar -jxf crewsoftware2014-03.tar.bz2, for windows use some kind of zip software)

Download the CREW SEVIRI database in common format

Go to a folder, where you have place to store a large SEVIRI dataset.
Download an unpack the CREW SEVIRI database in the common format
CREW4_SEVIRI_common_2008-06-13.tar.gz (9.9GB)
CREW4_SEVIRI_common_2008-06-17.tar.gz (2.7GB)
CREW4_SEVIRI_common_2008-06-18.tar.gz (1.9GB)
CREW4_SEVIRI_common_2008-06-22.tar.gz (3.0GB)
CREW4_SEVIRI_common_2008-07-03.tar.gz (3.0GB)
unpack the software archive (for Linux use tar -xzf CREW4_SEVIRI_common.tar.gz, for windows use some kind of zip software).
The day 2008-06-13 is the most complete one according to the submitted datasets. For the days 2008-06-17/2008-06-18 the core hours were during the night and around midnight. After extraction you got the following folders:
CREW4_SEVIRI_common/2008/MM/DD/HHmm/products/ where MM is the month, DD the day, HH the hour, mm the minute.
In these folders there are the data files in the common format with the name convention:
product-group-2008MMDDHHmm-MSG.png and .xml for example cth-GSF-200806131200-MSG.png.

Download the SEVIRI rgb images (and optional CREW SEVIRI database in the original format)

Go to a folder, where you have place to store another large dataset.
In order to plot RGB images you have to download following files:
CREW4_RGB_images.tar.gz (580MB)
After extraction you get the following folders:
CREW4_SEVIRI_source/2008/MM/DD/HHmm/products/ where MM is the month, DD the day, HH the hour, mm the minute.
In these folders there are the data files in the common format with the name convention:
product-group-2008MMDDHHmm-MSG.png and .xml for example cth-GSF-200806131200-MSG.png.

(Optional:
If it is necessary to convert the datasets from the original formats to the common one, you have to copy the complete folder of the group with the original dataset version from.
L2_CREW3_ORG
into the folder CREW4_SEVIRI_source. E.g. the folder DLR has to be here:
CREW4_SEVIRI_source/DLR

Adjust some files

A) Specify your idl start_up file to your <homedir>/.bashrc (or alternative shell start file)
(Note, that there might already be a idl start_up file defined, in this case don't change your .bashrc)
# set IDL startup file export IDL_STARTUP=${HOME}/.idl/idl_startup.pro

B) Add some lines in your idl_startup file ${HOME}/.idl/idl_startup.pro
(In case you dont have a idl start up file jet, create it. It does not matter where you put these lines in your IDL start up file.)

!PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/" !PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/subtools/" !PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/subtools/msg/"

DEFSYSV, '!PROJECTS', { $ crew_path: '<path to crewsoftware>/cws3/trunk/', $ crew_common_archive: '<path to SEVIRI database in common format>/CREW4_SEVIRI_common/', $ crew_source_archive: '<path to SEVIRI database in original format>/CREW4_SEVIRI_source/'}

where <path to crewsoftware> is the absolute path to the directory where you save the CREW software, e.g.
crew_path: '/tcc1/home/ulrich/crew/cws3/trunk/', $
<path to SEVIRI database in common format> is the path, where you extracted the CREW database in the common format and
<path to SEVIRI database in common format> is the path, where you extracted the CREW database in the original formats.

Create your first image

Start idl from your command line and type following commands:
o=obj_new('cws_read') o->set_date,2008,6,13,12,00 o->set_product,'ctt' o->set_group,'AWG' array = o->get_data() view2d,array,/cool,/colo,no_data_idx=where(array le 0) obj_destroy, o

or try another interactive example or some more advanced example

Installation procedure for comparing SEVIRI with ATRAIN measurements

Now you completed everything for inter-comparison of SEVIRI cloud retrievals from the common database. The CREW software in combination with the AVACS software has the functionality to compare the SEVIRI cloud retrievals with data sets of the ATRAIN satellite constellation. Therefore, you have to install also the AVACS software.

Get the AVACS software

Contact Phil Watts (EUMETSAT) for access to the AVACS software.

Download the dataset for AVACS

Go to a folder, where you have place to store a large AVACS dataset for CREW, e.g. AVACS_data.
We splitted the data into three packages:

levB.tar.bz2 (4.3 GB compressed / 8.3 GB extracted, Atrain data regridded to the SEVIRI grid)
levC.tar.bz2 (2.0 MB compressed / 14MB extracted, the CREW SEVIRI datasets along the Atrain track)
levS.tar.bz2 [optional] (76 GB compressed/ 139GB extracted, original CALIOP, CPR, and AMSR-E data files)

You can start downloading and unpacking levB and levC only. Untar the files with tar -jxf filename. (If you like to update the original satellite data and/or do another regridding, you can download levS and have a look at the folder structure.) After downloading all datasets the folder structure should look like this:
AVACS_data/levB/ AVACS_data/levC/ AVACS_data/levS/ [optional]

Installation of the AVACS software

In AVACS/version-x.y/DOCS there is a handbook for user. Follow the installation instruction in section 2 and 3.

Test new functionalities of the CREW/AVACS software

Try some CREW/AVACS examples

Updating of the level S / level B files [optional]

If there are new datasets available from CALIOP, CPR, AMSR-E or another validation dataset you may want to update the AVACS level S files and create new level B files. First you should have a look at the AVACS manual, how the files have to be stored within the AVACS levS folder. You can also download the levS.tar.bz2 files that contains the most recent version (status December 2013) of the validation data. Please have a look at the AVACS manual how to update level B files.

Updating of the level C files [optional]

In step 7 you downloaded ready-to-use level C files containing the data of the CREW SEVIRI algorithm along the ATRAIN track. If you update an SEVIRI dataset, you have to recreate the level C files.
o=obj_new('cws2') atrain_overpass=11317 o->create_level_c, atrain_overpass where atrain_overpass is the integer number of the Atrain orbit number, see CREW Data Set.

@@ Line 1: / Line 1: @@
-Within the CREW project a software package was developed for the inter-comparison and validation against CALIOP, CPR and AMSR-E data. The software package currently has a beta version status. We would like to invite every user to give us feedback to improve the software package. The functionality of the CREW software package includes:
+Several steps have to be completed before you can use the CREW software package if you like to intercompare SEVIRI retrievals:<br>
-* displaying cloud properties on the SEVIRI disk or other user specified areas
+== Installation procedure for SEVIRI dataset inter-comparison ==
-* histograms
+==='''Registration at ICARE''' <br>===
-* latitudinal means
+Please [mailto:%75%6c%72%69%63%68%2e%68%61%6d%61%6e%6e%40%6d%65%74%65%6f%73%77%69%73%73%2e%63%68;%6a%65%72%6f%6d%65%2e%72%69%65%64%69%40%75%6e%69%76%2d%6c%69%6c%6c%651%2e%66%72 send us an email] to register at the ICARE service.
-* plotting cloud property against viewing zenith angle
-* scatter plots (SEVIRI algorithm A against SEVIRI algorithm B)
-* scatter plots (SEVIRI algorithm against CALIOP/CPR/AMSR-E)
-* multi algorithms ensemble means and standard devations
-* ISCCP type plots (frequency distribution in dependence of cloud optical thickness and cloud top pressure)
-* cross section plots along the ATRAIN tracks
-* Taylor diagrams
-Several steps have to be completed before you can use the CREW software package if you like to intercompare SEVIRI retrievals:
+==='''Download the CREW software''' <br>===
-. Download and unpack the CREW software package
+Go to a folder, where you would like to install the CREW software (4.4MB).<br>
-. Download and unpack the CREW common database
+Download an unpack the CREW software package [ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/software/crewsoftware2014-03.tar.bz2 crewsoftware2014-03.tar.bz2]<br>
-. Adjust a few startup files
+unpack the software archive (for Linux use ''tar -jxf crewsoftware2014-03.tar.bz2'', for windows use some kind of zip software) <br>
-If you like to compare SEVIRI against Atrain sensors, following packages has to be installed additionally:
-. Download the AVACS software
-. Create a database of reference datasets
-. Use the CREW and AVACS software package in combination
-. Download an unpack the CREW software package
+==='''Download the CREW SEVIRI database in common format''' <br>===
-[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/software/crewsoftware.tar.gz]
+Go to a folder, where you have place to store a large SEVIRI dataset.<br>
+Download an unpack the CREW SEVIRI database in the common format <br>
+<!--[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common.tar.gz CREW4_SEVIRI_common.tar.gz]-->
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common_2008-06-13.tar.gz CREW4_SEVIRI_common_2008-06-13.tar.gz] (9.9GB)<br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common_2008-06-17.tar.gz CREW4_SEVIRI_common_2008-06-17.tar.gz] (2.7GB)<br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common_2008-06-18.tar.gz CREW4_SEVIRI_common_2008-06-18.tar.gz] (1.9GB)<br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common_2008-06-22.tar.gz CREW4_SEVIRI_common_2008-06-22.tar.gz] (3.0GB)<br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_SEVIRI_common_2008-07-03.tar.gz CREW4_SEVIRI_common_2008-07-03.tar.gz] (3.0GB)<br>
+unpack the software archive (for Linux use ''tar -xzf CREW4_SEVIRI_common.tar.gz'', for windows use some kind of zip software). <br>
+The day 2008-06-13 is the most complete one according to the submitted datasets. For the days 2008-06-17/2008-06-18 the core hours were during the night and around midnight.
+After extraction you got the following folders: <br>
+CREW4_SEVIRI_common/2008/''MM''/''DD''/''HHmm''/products/ where ''MM'' is the month, ''DD'' the day, ''HH'' the hour, ''mm'' the minute.<br>
+In these folders there are the data files in the common format with the name convention: <br>
+''product''-''group''-2008''MMDDHHmm''-MSG.png and .xml for example cth-GSF-200806131200-MSG.png.
+==='''Download the SEVIRI rgb images (and optional CREW SEVIRI database in the original format)''' <br>===
+Go to a folder, where you have place to store another large dataset.<br>
+In order to plot RGB images you have to download following files: <br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/CREW4_RGB_images.tar.gz CREW4_RGB_images.tar.gz] (580MB)<br>
+After extraction you get the following folders: <br>
+CREW4_SEVIRI_source/2008/''MM''/''DD''/''HHmm''/products/ where ''MM'' is the month, ''DD'' the day, ''HH'' the hour, ''mm'' the minute.<br>
+In these folders there are the data files in the common format with the name convention: <br>
+''product''-''group''-2008''MMDDHHmm''-MSG.png and .xml for example cth-GSF-200806131200-MSG.png.<br>
+<br>
+(Optional:<br>
+If it is necessary to convert the datasets from the original formats to the common one,
+you have to copy the complete folder of the group with the original dataset version from.<br>
+[ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2011_CREW3/L2_CREW3_ORG/ L2_CREW3_ORG]<br>
+into the folder ''CREW4_SEVIRI_source''. E.g. the folder DLR has to be here:<br>
+CREW4_SEVIRI_source/DLR
+==='''Adjust some files'''<br>===
+A) Specify your idl start_up file to your <homedir>/.bashrc (or alternative shell start file)<br>
+(Note, that there might already be a idl start_up file defined, in this case don't change your .bashrc)<br>
+<code># set IDL startup file <br>
+export IDL_STARTUP=${HOME}/.idl/idl_startup.pro</code>
+B) Add some lines in your idl_startup file ${HOME}/.idl/idl_startup.pro <br>
+(In case you dont have a idl start up file jet, create it. It does not matter where you put these lines in your IDL start up file.)
+<code>!PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/"<br>
+!PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/subtools/"<br>
+!PATH=!PATH+":<path to crewsoftware>/cws3/trunk/tools/subtools/msg/"</code><br>
+<code>DEFSYSV, '!PROJECTS', { $<br>
+crew_path:    '<path to crewsoftware>/cws3/trunk/', $<br>
+crew_common_archive: '<path to SEVIRI database in common format>/CREW4_SEVIRI_common/', $<br>
+crew_source_archive: '<path to SEVIRI database in original format>/CREW4_SEVIRI_source/'}</code>
+where <code><path to crewsoftware></code> is the absolute path to the directory where you save the CREW software, e.g.<br>
+crew_path:    '/tcc1/home/ulrich/crew/cws3/trunk/', $<br>
+<code><path to SEVIRI database in common format></code> is the path, where you extracted the CREW database in the common format and <br>
+<code><path to SEVIRI database in common format></code> is the path, where you extracted the CREW database in the original formats.
+==='''Create your first image'''<br>===
+Start idl from your command line and type following commands:<br>
+<code>o=obj_new('cws_read')<br>
+o->set_date,2008,6,13,12,00<br>
+o->set_product,'ctt'<br>
+o->set_group,'AWG'<br>
+array = o->get_data()<br>
+view2d,array,/cool,/colo,no_data_idx=where(array le 0)<br>
+obj_destroy, o<br>
+</code>
+or try another [[interactive example]]
+or some [[more advanced example]]
+== Installation procedure for comparing SEVIRI with ATRAIN measurements ==
+Now you completed everything for inter-comparison of SEVIRI cloud retrievals from the common database. The CREW software in combination with the AVACS software has the functionality to compare the SEVIRI cloud retrievals with data sets of the ATRAIN satellite constellation. Therefore, you have to install also the AVACS software.
+==='''Get the AVACS software'''<br>===
+Contact [mailto:&#x70;&#x68;&#x69;&#x6c;&#x69;&#x70;&#x2e;&#x77;&#x61;&#x74;&#x74;&#x73;&#x40;&#x65;&#x75;&#x6d;&#x65;&#x74;&#x73;&#x61;&#x74;&#x2e;&#x69;&#x6e;&#x74; Phil Watts (EUMETSAT)]   for access to the AVACS software.
+==='''Download the dataset for AVACS'''<br>===
+Go to a folder, where you have place to store a large AVACS dataset for CREW, e.g. <code>''AVACS_data''</code>.<br>
+We splitted the data into three packages:
+* [ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/AVACS_data/levB.tar.bz2 levB.tar.bz2] (4.3 GB compressed / 8.3 GB extracted, Atrain data regridded to the SEVIRI grid)
+* [ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/AVACS_data/levC.tar.bz2 levC.tar.bz2] (2.0 MB compressed / 14MB extracted, the CREW SEVIRI datasets along the Atrain track)
+* [ftp://ftp.icare.univ-lille1.fr/DATA/FS117/crew/data/data_2014_CREW4/AVACS_data/levS.tar.bz2 levS.tar.bz2] [optional] (76 GB compressed/ 139GB extracted, original CALIOP, CPR, and AMSR-E data files)
+You can start downloading and unpacking levB and levC only.
+Untar the files with <code>tar -jxf filename</code>.
+(If you like to update the original satellite data and/or do another regridding, you can download levS and have a look at the folder structure.) After downloading all datasets the folder structure should look like this:<br>
+<code>
+''AVACS_data''/levB/<br>
+''AVACS_data''/levC/<br>
+''AVACS_data''/levS/  [optional]<br>
+</code>
+==='''Installation of the AVACS software'''<br>===
+In ''AVACS/version-x.y/DOCS'' there is a handbook for user. Follow the installation instruction in section 2 and 3.
+==='''Test new functionalities of the CREW/AVACS software'''<br>===
+Try some [[CREW/AVACS examples]]
+==='''Updating of the level S / level B  files [optional]'''<br>===
+If there are new datasets available from CALIOP, CPR, AMSR-E or another validation dataset you may want to update the AVACS level S files and create new level B files.
+First you should have a look at the AVACS manual, how the files have to be stored within the AVACS levS folder. You can also download the levS.tar.bz2 files that contains the most recent version (status December 2013) of the validation data. Please have a look at the AVACS manual how to update level B files.
+==='''Updating of the level C files [optional]'''<br>===
+In step 7 you downloaded ready-to-use level C files containing the data of the CREW SEVIRI algorithm along the ATRAIN track. If you update an SEVIRI dataset, you have to recreate the level C files. <br>
+<code>
+o=obj_new('cws2')<br>
+atrain_overpass=11317<br>
+o->create_level_c, atrain_overpass<br>
+</code>
+where <code>atrain_overpass</code> is the integer number of the Atrain orbit number, see [[CREW_Data_Set| CREW Data Set]].

Installing the CREW software package - CrewWiki