Project

General

Profile

segmentation fault with lowpass/highpass filtering

Added by Paolo Davini almost 10 years ago

Hi all,

I am trying to use lowpass filtering with CDO, although I am having issues and frankly I do not understand why.
I have a timeseries of 142 year of SST anomalies. Just one grid point. When I apply the filter, I always get this. Same problem with highpass.
I also tried with different files or different t-axis but apparently it does not change the result.

cdo -v lowpass,10 global_sst_anomaly.nc try.nc

time step 1 year
Segmentation fault (core dumped)

The time properties of the file seems pretty normal.
cdo sinfo global_sst_anomaly.nc

  File format: netCDF
    -1 : Institut Source   Ttype    Levels Num  Gridsize Num Dtype : Parameter ID
     1 : unknown  HadISST  instant       1   1         1   1  F32  : -1            
   Grid coordinates :
     1 : lonlat       > size      : dim = 1  nx = 1  ny = 1
                        lon       : first = 0  degrees_east
                        lat       : first = 0  degrees_north
   Vertical coordinates :
     1 : surface                  : 0 
   Time coordinate :  142 steps
     RefTime =  0001-01-01 00:00:00  Units = hours  Calendar = standard  Bounds = true
  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss  YYYY-MM-DD hh:mm:ss
  1871-12-01 00:00:00  1872-12-01 00:00:00  1873-12-01 00:00:00  1874-12-01 00:00:00
  1875-12-01 00:00:00  1876-12-01 00:00:00  ......               2012-12-01 00:00:00
cdo sinfo: Processed 1 variable over 142 timesteps ( 0.00s )

My version is 1.6.2
cdo -V

Features: PTHREADS NC4 SZ Z JASPER UDUNITS2 PROJ.4
Libraries: PTHREADS NC4 SZ zlib/1.2.3 JASPER UDUNITS2 PROJ.4
Filetypes: srv ext ieg grb grb2 nc nc2 nc4 nc4c 
     CDI library version : 1.6.2 of Sep 18 2013 13:29:52
 CGRIBEX library version : 1.6.2 of Jul 18 2013 09:34:47
GRIB_API library version : 1.9.16
  netCDF library version : 4.2 of Sep 18 2013 11:08:14 $
    HDF5 library version : 1.8.8
 SERVICE library version : 1.3.1 of Sep 18 2013 13:29:33
   EXTRA library version : 1.3.1 of Sep 18 2013 13:29:24
     IEG library version : 1.3.1 of Sep 18 2013 13:29:28
    FILE library version : 1.8.2 of Sep 18 2013 13:29:24

What am I missing? Does anyone has ever experience something similar?
Many thanks for any replies!
Cheers
Paolo


Replies (5)

RE: segmentation fault with lowpass/highpass filtering - Added by Uwe Schulzweida almost 10 years ago

Hi Paolo,

Thanks for this report! This is definitely a bug. An array is initialized out of bounds.
This bug will be fixed in the next CDO release. A CDO prerelase which contains the bug fix is available in https://code.zmaw.de/projects/cdo/files

Cheers,
Uwe

RE: segmentation fault with lowpass/highpass filtering - Added by Paolo Davini almost 10 years ago

Dear Uwe,

thank you so much for the prompt reply (and solution)!

I successfully installed the new cdo version and actually I no longer get any error, so I guess the modification works!
Unfortunately, I still have some problem with the lowpass filtering.

When I do:

cdo -v lowpass,10 global_sst_anomaly.nc try.nc

 OpenMP:  num_procs = 24  max_threads = 1
cdo lowpass: Using fftw lib
cdo lowpass: Time step 1 year
cdo lowpass: Processed 142 values from 1 variable over 142 timesteps ( 0.00s )

and it successfully produce a new file. However, if I do:

cdo diff global_sst_anomaly.nc try.nc

  0 of 142 records differ
cdo diff: Processed 284 values from 2 variables over 284 timesteps ( 0.00s )

Which actually means that the lowpass operation has not changed the file, but just copied it.
The files is the same as in the previous post, with yearly timesteps.

the version is slightly different because is compiled locally.

cdo -V

Climate Data Operators version 1.6.4rc5 (http://code.zmaw.de/projects/cdo)
Compiled: by paolo on sansone (x86_64-unknown-linux-gnu) Apr 28 2014 11:02:58
Compiler: gcc -std=gnu99 -g -O2 -fopenmp 
 version: gcc (Ubuntu/Linaro 4.7.3-1ubuntu1) 4.7.3
Features: PTHREADS OpenMP NC4 Z FFTW3
Libraries:
Filetypes: srv ext ieg grb nc nc2 nc4 nc4c 
     CDI library version : 1.6.4rc3 of Apr 28 2014 11:02:49
 CGRIBEX library version : 1.6.4 of Mar 28 2014 13:24:51
  netCDF library version : 4.2.1.1 of Apr 28 2014 10:55:03 $
 SERVICE library version : 1.3.2 of Apr 28 2014 11:02:33
   EXTRA library version : 1.3.2 of Apr 28 2014 11:02:26
     IEG library version : 1.3.2 of Apr 28 2014 11:02:31
    FILE library version : 1.8.2 of Apr 28 2014 11:02:27

I attached the original file just in case you want to double check it by yourself.
Am I misunderstanding something about the usage of the cdo lowpass filtering?
Thank you once more
Cheers
Paolo

RE: segmentation fault with lowpass/highpass filtering - Added by Uwe Schulzweida almost 10 years ago

Dear Paolo,

I can reproduce this problem! The result seems to be wrong for me. But I have no idea where the problem comes from.
The author of this module is not at the institute at the moment. He will be back in two weeks.

Cheers,
Uwe

Parameters of Low/Highpass Filter - Added by Cedrick Ansorge almost 10 years ago

Dear Paolo,

the parameter you pass to the lowpass operator is a frequency per year. So, if you use lowpass,10 you ask for a filter that passes all frequencies smaller than 1/10 of a year, which in your case is the entire signal.

I think, what you are looking for is cdo lowpass,0.1 which would filter out all frequencies with a period small than ten years. We checked this with your time series, and the result looks as expected. Just one remark: do not forget to detrend the time series before filtering, otherwise you will get an artificial signal forcing the data to periodicity at the beginning and end of the time series.

So, my recommendation is

cdo lowpass,0.1 -detrend ts.nc ts_filt_10year.nc

Best Regards,
Cedrick

RE: segmentation fault with lowpass/highpass filtering - Added by Ulf Graewe 4 months ago

Dear cdo wizards,
the segmentation fault issue still persists.

cdo --no_history -O  lowpass,26.0714 ssh.nc filling.nc
Segmentation fault (core dumped)

It works if I do it with a time series. However, as soon as I go full 3d I get a segmentation fault.

I tried with:

cdo -V
Climate Data Operators version 2.2.2 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -std=gnu++17 -g -O2 -fopenmp -pthread
CXX version : g++ (SUSE Linux) 7.5.0
C Compiler: gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (SUSE Linux) 7.5.0
F77 Compiler: gfortran -g -O2
F77 version : GNU Fortran (SUSE Linux) 7.5.0
Features: 1007GB 16threads c++17 OpenMP45 Fortran pthreads HDF5 NC4/HDF5/threadsafe OPeNDAP fftw3 hirlam_extensions sse2
Libraries: yac/3.0.1 NetCDF/4.9.1-rc1 HDF5/1.12.1
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 nc1 nc2 nc4 nc4c nc5 nczarr 
     CDI library version : 2.2.4
 cgribex library version : 2.1.1
  NetCDF library version : 4.9.1-rc1 of Oct 31 2022 09:54:55 $
    exse library version : 1.5.0
    FILE library version : 1.9.1

but also with:

cdo -V
Climate Data Operators version 2.3.0 (https://mpimet.mpg.de/cdo)
System: x86_64-pc-linux-gnu
CXX Compiler: g++ -std=gnu++17 -g -O2 -fopenmp -pthread
CXX version : g++ (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
C Compiler: gcc -g -O2 -fopenmp -pthread -pthread
C version : gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
F77 Compiler: f77 -g -O2
F77 version : GNU Fortran (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Features: 15GB 4threads c++17 OpenMP45 Fortran pthreads NC4/HDF5 OPeNDAP sse2
Libraries: yac/3.0.1 NetCDF/4.7.3
CDI data types: SizeType=size_t
CDI file types: srv ext ieg grb1 nc1 nc2 nc4 nc4c nc5 
     CDI library version : 2.3.0
 cgribex library version : 2.1.1
  NetCDF library version : 4.7.3 of Jan 23 2020 04:18:16 $
    exse library version : 1.5.0
    FILE library version : 1.9.1

    (1-5/5)