converting regular lon lat into an index

Hallo,

We have some rather large netCDF files, which we would like to reduce in size, by only keeping data for grid points where data are present.

So from a netCDF structure:

dimensions:
    lon = 720 ;
    lat = 360; 
    time = 72635 ;
variables:
    double lon(lon) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
    double lat(lat) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
    double time(time) ;
        time:standard_name = "time" ;
        time:units = "days since 1860-01-01 00:00:00" ;
        time:calendar = "standard" ;
    float tasAdjust(lat, lon, time) ;
        tasAdjust:standard_name = "air_temperature" ;
        tasAdjust:long_name = "Bias-Corrected Near-Surface Air Temperature" ;
        tasAdjust:units = "K" ;
        tasAdjust:_FillValue = 1.e+20f ;
        tasAdjust:missing_value = 1.e+20f ;

we would like to get to a structure as follows, which only contains data for valid grid points, basically 59191 lon lat pairs:

dimensions:
    index = 59191;
    time = 72635 ;
variables:
    double lon(index) ;
        lon:standard_name = "longitude" ;
        lon:long_name = "longitude" ;
        lon:units = "degrees_east" ;
        lon:axis = "X" ;
    double lat(index) ;
        lat:standard_name = "latitude" ;
        lat:long_name = "latitude" ;
        lat:units = "degrees_north" ;
        lat:axis = "Y" ;
    double time(time) ;
        time:standard_name = "time" ;
        time:units = "days since 1860-01-01 00:00:00" ;
        time:calendar = "standard" ;
    float tasAdjust(index, time) ;
        tasAdjust:standard_name = "air_temperature" ;
        tasAdjust:long_name = "Bias-Corrected Near-Surface Air Temperature" ;
        tasAdjust:units = "K" ;
        tasAdjust:_FillValue = 1.e+20f ;
        tasAdjust:missing_value = 1.e+20f ;

is there a simple cdo command that can do that?
If I do a brute force conversion with a R-script, it takes 30 seconds per gridcell, so it would take nearly 20 hours to convert one file.

cheers
Peter

Replies (5)

RE: converting regular lon lat into an index - Added by Ralf Mueller about 8 years ago

Hi Peter!
sorry there is no simple command. Here are my comments:

if filesize is a problem, you should thing about compression: using nc4 together with zip-compression can be really powerfull
```
cdo -f nc4 -z zip_1 <ifile> <ofile>
```
This type of compression part of the netcdf4 standard
One drawback I see in you target output it the lack of geometric information: You target grid is no longer global, hence, the computation of cell bounds and area is not given. In order to compute horizontal means or other intergrals, you would have to compute the bounds manually and add them to the output file.

I would try something like this:

identidy the locations, that will be part of the new reduced grid
create a netcdf file with the appropriate dimensions and a data variable based on that grid/dimensions
apply nearest neighbour interpolation from yur input file towards the target grid
```
cdo -remapnn,targetGrid.nc input.nc <ofile>
```

hth
ralf

RE: converting regular lon lat into an index - Added by P. Anthoni about 8 years ago

Hi Ralf,

thanks for the info. Our files are already nc4 and compressed with zip9, that really shrunk them down from 70GByte to like 14Gbyte.
We would like to convert to the index format, since our model hopefully would be able to read the files faster that way.
But I just read, that one of the effects of the zip compression is that the storage needed for the missing data is reduced significantly.
I guess that means the size would not get much smaller, if we would change to an index format.
I might try my brute force R-script on one file and see, how much smaller it gets compared to the zipped version.

many thanks for your quick response.

cheers
Peter

RE: converting regular lon lat into an index - Added by Ralf Mueller about 8 years ago

There is a new operator called reduce. It basically works like you wanted, but it adds coordinate bounds. Would you like to test it?

RE: converting regular lon lat into an index - Added by Joerg Steinkamp about 7 years ago

Dear Ralf and Peter,

Quiet a while ago, I wrote a crude python script [[https://github.com/joergsteinkamp/wfd4lpj]], doing that for a specific climate driver. I think I can thrash that now and switch to 'cdo reducegrid', thanks.

However, I found your thread while searching for the opposite function: How to convert NetCDF files from the reduced grid, as described here back to a regular grid? I sadly didn't find it on the cdo help page. Do you have a solution for it? Or should I open a new thread for my question?

Thanks & kind regards
Jörg

RE: converting regular lon lat into an index - Added by Ralf Mueller about 7 years ago

Hi Joerg!
Regular grids are built-in (see here). If you want to enlarge your local grid to that, just use your favorite remapping operator.

hth
ralf

(1-5/5)

Project

General

Profile

CDO