Project

General

Profile

Generating NetCDF File from Text File (Errors included)

Added by Aaron Perry almost 8 years ago

Hi All,

I have an enormous temperature dataset that I am trying to use to create a NetCDF file. I am using the following code in my terminal to try and generate the NC file but,m I am running into issues:

CODE:
cdo -f nc input,grid 1234.nc < /home/weather/tmaxs

Sample Data (tmaxs):
36.68167 -101.50528 98.6218
36.2967 -99.7689 97
35.3568 -99.2042 97.1855
34.9894 -99.0525 97.4309
34.21 -98.59 98.1072
34.5584 -98.4172 96.7855
35.8517 -97.4142 96.575
36.7683 -96.0261 94.0055
34.8822 -95.783 93.7527
35.65667 -95.36139 93.7509

grid file:
gridtype = latlon
xsize = 10
ysize = 10
xvals = -101.50528 -99.7689 -99.2042 -99.0525 -98.59 -98.4172 -97.4142 -96.0261 -95.783 -95.36139
yvals = 36.68167 36.2967 35.3568 34.9894 34.21 34.5584 35.8517 36.7683 34.8822 35.65667

The error:
@cdo input: Enter all 100 elements of record 1!

cdo input (Abort): Too few input elements (30 of 100)!@


Replies (18)

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

Your xsize, and ysize are 10, so you need 10x10=100 data points but only found 10 in your sample data! Also, since you already describe the input grid info, you only need the 3rd column of tmax - no need of lon/lat columns. For example, try with xsize=3 and ysize =3, and use only 9 data points in your sample data (only the 3rd colum) it'll work.

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

Hi JT,

Thanks for the reply!

I changed the x/y sizes to 3 and then used only the third column (tmaxs) and deleted the last line to get 9 entries.

I am now getting the following error:

cdo input: Enter all 9 elements of record 1! cdo input: Enter all 9 elements of record 2! cdo input: ( 0.00s )

tmaxs_only (66 Bytes) tmaxs_only
1234.nc (968 Bytes) 1234.nc
grid (205 Bytes) grid
tmaxs (244 Bytes) tmaxs

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

That is not an error message - it's just a message - you can suppress it by -s option.

with your input data:

cdo -f nc input,grid 1234.nc < tmaxs_only 
cdo input: Enter all 9 elements of record 1!
cdo input: Enter all 9 elements of record 2!
cdo input:  ( 0.00s )
ncdump -h 1234.nc 
netcdf \1234 {
dimensions:
        lon = 3 ;
        lat = 3 ;
        time = UNLIMITED ; // (1 currently)
variables:
        double lon(lon) ;
                lon:standard_name = "longitude" ;
                lon:long_name = "longitude" ;
                lon:units = "degrees_east" ;
                lon:axis = "X" ;
        double lat(lat) ;
                lat:standard_name = "latitude" ;
                lat:long_name = "latitude" ;
                lat:units = "degrees_north" ;
                lat:axis = "Y" ;
        double time(time) ;
                time:standard_name = "time" ;
                time:units = "hours since 0-0-0 00:00:00" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
        float var1(time, lat, lon) ;

// global attributes:
                :CDI = "Climate Data Interface version 1.7.1 (http://mpimet.mpg.de/cdi)" ;
                :Conventions = "CF-1.4" ;
                :history = "Tue Jun 21 10:37:32 2016: cdo -f nc input,grid 1234.nc" ;
                :CDO = "Climate Data Operators version 1.7.1 (http://mpimet.mpg.de/cdo)" ;
}

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

Ok, great!

I have opened the .nc file using NASA's Panoply Data Viewer and I am not seeing any temperature data being plotted.

Plot:

What is the next step?

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

here is my Panoply image of 1234.nc

1234.png (24.1 KB) 1234.png

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

it is plotted somewhere on your map (scale on the colorbar suggests, Panoply read the data). Also, try with ncview

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

I am trying to verify that the data is being correctly displayed/plotted so, I have increased the number of data points to 49. FWIW, I have setup the latitudes (yvals) in ascending order from lowest to highest.

grid2 file:
gridtype = latlon
xsize = 7
ysize = 7
xvals = -97.423 -97.65417 -97.31667 -97.43333 -98.02472 -97.5122 -99.21833 -95.46194 -94.8592 -99.1742 -100.927 -95.24167 -102.21306 -97.6799 -97.7604 -95.41389 -96.36472 -98.2354 -102.91667 -106.37583 -96.39889 -99.6822 -97.09389 -96.86806 -97.36139 -96.8555 -97.0189 -97.31806 -97.19889 -98.59 -100.2831 -98.4172 -95.783 -99.0525 -101.7042 -99.2042 -97.6006 -97.64694 -95.36139 -101.395 -97.4142 -102.55 -95.98444 -97.0894 -95.8872 -99.7689 -101.50528 -97.10194 -96.0261
yvals = 25.9141 26.22806 27.63333 27.7 27.74111 27.7742 28.45667 29.10972 29.2733 29.3601 29.3784 29.51889 30.04806 30.1831 30.3208 30.35667 30.58917 30.7406 30.91194 31.81111 32.03111 32.4105 32.66361 32.68083 32.81917 32.8519 32.8978 32.97333 33.20611 34.21 34.4272 34.5584 34.8822 34.9894 35.2295 35.3568 35.3889 35.53417 35.65667 35.695 35.8517 36.0167 36.03944 36.1624 36.1994 36.2967 36.68167 36.73667 36.7683

Temperature Data File:
91.8564
93.5891
89.6
91.4
95.2418
93.2945
98.6782
90.0436
88.7691
95.0236
97.0564
90.5545
96.14
94.0636
94.48
91.1109
92.0327
93.0909
93.7491
95.8927
92.8818
93.9364
94.9127
94.6764
95.7218
95.7982
96.1509
95.5491
95.4345
98.1072
97.4145
96.7855
93.7527
97.4309
94.4436
97.1855
94.52
94.98
93.7509
96.8927
96.575
96.3327
95.8564
95.5
95.0527
97
98.6218
96.6836
94.0055

Additionally, I've tried exporting the data as GRIB2.

tx_out.grb2 (282 Bytes) tx_out.grb2
grid2 (927 Bytes) grid2
d0_tx_tmaxs (369 Bytes) d0_tx_tmaxs
d0_tx (1.21 KB) d0_tx

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

I'm starting to think that the lats/lons need to be ascending/descending in a certain order...

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

Hi,

Is your data on a regular lon-lat grid?. What is the resolution of lon/lat? If not then it is better to use the gridtype "unstructured" or "cell" [instead of "lonlat"] for creating the netcdf. However, you won't be able to visualize it directly using panoply/ncview, but can be visualized after interpolating to a regular lon-lat grid OR use tools like Generic Mapping tools [GMT]/Python/Matlab etc for directly visualizing irregular/cell grids [polygon shade/fill].

Hope this helps.
J

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

My data is in text format so, I'm guessing regular lat-lon gridding... As for the resolution, the distance between data points is random. Essentially, I'm looking for/asking if there is a way to convert data in lat,lon,temp csv format to a netcdf file, even if this requires interpolating between the lat/lons.

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

here is an example, to interpolate to regular grid:

cdo -f nc input,org.grd new_1234.nc < dat
cdo -f nc remapdis,interp.txt new_1234.nc interpolated.nc

Please have a look at the attached files.

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

WOW. This is fantastic! Thank you so much for the help J! I am going to play around with the interp.txt file and then I will post the results back here. Your guidance has been truly invaluable!

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

Additionally, I am using this plot that I generated using matplotlib/python for comparison:

tx.png (218 KB) tx.png

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

I have been able to replicate (more or less) what you suggested J. It still looks slightly off from my other image though. Are you aware of any other ways to get this plot correctly? Should I be defining X/Y bounds at all?

RE: Generating NetCDF File from Text File (Errors included) - Added by Jaison-Thomas Ambadan almost 8 years ago

It still looks slightly off from my other image though

it depends on resolution & interpolation - I think the matplotlib/basemap default is cubic spline (?). Anyway, in the example [interp.txt], I used a coarse resolution of 1.5degree. You could try changing that - depending on the max/min of lon/lat values, for e.g., try:

gridtype  = lonlat
gridsize  = 2116
xname     = lon
xlongname = longitude
xunits    = degrees_east
yname     = lat
ylongname = latitude
yunits    = degrees_north
xsize     = 46
ysize     = 46
xfirst    = -106.375
xinc      = 0.25
yfirst    = 25.914
yinc      = 0.25

PS: if you can plot the original data with Python, then I guess you don't need to do all these stuff :)

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

While that is true, my end-goal is to transfer my simple text data to a more standardized, common format, such as NetCDF or GRIB2. Also, the changes you suggested with the interp.txt file has improved both the location and resolution of the data!

RE: Generating NetCDF File from Text File (Errors included) - Added by Aaron Perry almost 8 years ago

The data appears to be plotting better!

I went in and added a few more data points to expand the domain a little bit and it appears that I have inadvertently forced the X/Y bounds too far north and east, beyond TX and OK... Whoops.

RE: Generating NetCDF File from Text File (Errors included) - Added by Miles Sowden over 3 years ago

Jaison-Thomas Ambadan wrote:

Hi,

Is your data on a regular lon-lat grid?. What is the resolution of lon/lat? If not then it is better to use the gridtype "unstructured" or "cell" [instead of "lonlat"] for creating the netcdf. However, you won't be able to visualize it directly using panoply/ncview, but can be visualized after interpolating to a regular lon-lat grid OR use tools like Generic Mapping tools [GMT]/Python/Matlab etc for directly visualizing irregular/cell grids [polygon shade/fill].

Hope this helps.
J

Hi Jaison, Great answer.

How would you change it if it was an irregular grid of 49 monitored sites (x,y) and temperature?
And a second complication if we added time to the mix (hourly temperature at each site for a year).

    (1-18/18)