Add a reader for EUMETSAT HSAF SEVIRI H60 and H63 #3287

albertbrotzer · 2025-11-10T10:38:03Z

The H SAF product owners recently changed the output format of these products, and EUMETSAT requested support for the new format in EUMETView.

Closes Add a reader for EUMETSAT HSAF SEVIRI H60 and H63 #3282
Tests added - not sure id this is sufficient
Fully documented
Add your name to AUTHORS.md if not there already

ameraner

Thanks for this! I would propose to actually have only one yaml file for this reader, and you can have different file types inside the same yaml (see e.g. the other PR here https://github.com/pytroll/satpy/pull/2793/files#diff-4ce2f1c002c4dae06506b9ad57f5a0504f278028002abc6263d2cd90ae50c8c6 ). This way it's more user friendly as there is only one reader for all HSAF products, and it's easier to maintain.

Also, there are a few things to fix from the pre-commit checks.

satpy/tests/enhancement_tests/test_hsaf.py

satpy/readers/hsaf_nc.py

ameraner · 2025-11-10T12:02:31Z

Also, would it be possible to add support for the H90 product as well, if it's not a lot of work? That way we could close the other PR #2793 for good

albertbrotzer · 2025-11-12T06:03:40Z

Hi @ameraner, there’s now support for H90 products. It uses the same reader and enhancements as H60 and H63. Please let us know if there’s anything else missing. Thanks for the support!

codecov · 2025-11-12T09:31:26Z

Codecov Report

❌ Patch coverage is 98.08917% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 96.33%. Comparing base (ee989f6) to head (cba3797).
⚠️ Report is 5 commits behind head on main.

Files with missing lines	Patch %	Lines
satpy/readers/hsaf_nc.py	95.16%	3 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #3287    +/-   ##
========================================
  Coverage   96.33%   96.33%            
========================================
  Files         463      466     +3     
  Lines       58870    59036   +166     
========================================
+ Hits        56711    56874   +163     
- Misses       2159     2162     +3

Flag	Coverage Δ
behaviourtests	`3.59% <0.00%> (-0.02%)`	⬇️
unittests	`96.42% <98.08%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

ameraner

Thank you for the updates! I took a closer looks and gave some more comments. The area definition assignments needs to be revised, otherwise only minor things.

satpy/readers/hsaf_nc.py

satpy/etc/readers/hsaf_nc.yaml

satpy/readers/hsaf_nc.py

ameraner · 2025-11-12T13:18:49Z

Also, seems that the gzip decompression things are not well coverd by tests, according to codecov...
Note that for some readers we delegate the decompression to the user using FSFile, e.g. see

satpy/satpy/readers/hrit_jma.py

Lines 74 to 90 in 3e2edf2

    
           Compression 
        
           ----------- 
        
           Gzip-compressed MTSAT files can be decompressed on the fly using 
        
           :class:`~satpy.readers.core.remote.FSFile`: 
        
           .. code-block:: python 
        
               import fsspec 
        
               from satpy import Scene 
        
               from satpy.readers.core.remote import FSFile 
        
               filename = "/data/HRIT_MTSAT1_20090101_0630_DK01IR1.gz" 
        
               open_file = fsspec.open(filename, compression="gzip") 
        
               fs_file = FSFile(open_file) 
        
               scn = Scene([fs_file], reader="jami_hrit") 
        
               scn.load(["IR1"])

maybe the same could be done here for simplicity?

ameraner · 2025-11-12T20:33:35Z

Actually, @mraspaud pointed me to the generic_open context manager, that by using Upath it could actually handle the gz files automagically without extra code, see below (it's already used in a few readers):

satpy/satpy/readers/core/utils.py

Lines 346 to 350 in 6cdfea4

    
           def generic_open(filename, *args, **kwargs): 
        
               """Context manager for opening either a regular file or a compressed file. 
        
               Yields an open file-like object. 
        
               """

This commit introduces a reader enhancement for HSAF precipitation products h60b, h63 and h90. The reader includes handling for externally compressed (.gz) files by automatically unzipping them to a local temporary location.

This commit restructures and improves the reader following initial review feedback. Key changes: - Unified all file_type definitions into a single YAML configuration file - Simplified and cleaned up reader logic, removing redundant code - Expanded and clarified docstrings across the reader implementation - Improved handling of area definitions in the HSAF NetCDF reader - Generalized dataset definitions so they can be reused by multiple file types

Replaced the custom logic for handling externally zipped NetCDF files with Satpy's generic_open helper, which transparently supports both compressed and uncompressed inputs. This simplifies the reader implementation and removes compression-detection code. Because generic_open uses a context manager and closes the file as soon as the reader exits the `with` block, the dataset must now be fully loaded immediately. The reader therefore calls `.compute()` on the xarray dataset to ensure all data is available after the context closes.

albertbrotzer · 2025-11-17T14:10:01Z

@ameraner @mraspaud the PR should be ready for review again. I think we addressed all comments and checks. Thanks again for your exceptional support and feedback!

ameraner

Thanks for the updates! Looks good - only one point on the use of compute, and a minor point on the attributes

satpy/readers/hsaf_nc.py

…mains accessible - Replace the short-lived with generic_open(...) usage with manual __enter__() / __exit__() handling - Avoid loading the entire dataset via .compute(), preserving lazy operation while preventing “seek of closed file” errors. - Ensure the NetCDF file remains open for Dask-backed arrays until the reader is destroyed. - Add safe cleanup in __del__() using explicit context-manager exit and catching expected cleanup-related exceptions.

...as they're added automatically through the property methods

ameraner

LGTM, thanks for taking the initiative to add this reader and all the updates!

ReadTheDocs is currently failing, but only due to a warning regarding not being able to fetch a numpy inventory due to HTTP error, so probably just a glitch and nothing related to the PR.

pnuu · 2025-11-19T15:23:00Z

There are some failing tests. I think the others are glitches, but I think this I fixed in an earlier PR that was merged:

FAILED satpy/tests/test_utils.py::TestGetSatPos::test_get_satpos_from_satname - ImportError: cannot import name 'tlefile' from 'pyorbital.orbital' (/home/runner/miniconda3/envs/test-environment/lib/python3.11/site-packages/pyorbital/orbital.py)

The same error is in few other tests, but all of them hopefully go away if you merge the current main branch to your PR.

pnuu · 2025-11-19T15:26:30Z

Ah, the fix is still unmerged in #3294

albertbrotzer · 2025-11-19T16:13:39Z

LGTM, thanks for taking the initiative to add this reader and all the updates!

ReadTheDocs is currently failing, but only due to a warning regarding not being able to fetch a numpy inventory due to HTTP error, so probably just a glitch and nothing related to the PR.

Thanks to you all for supporting us in the process. I think we can learn a lot from you guys and we will for sure add more as soon as time allows.

pnuu · 2025-11-19T20:29:51Z

Ok, we merged couple of PRs to main and main to your PR branch. Hopefully now all the tests will pass.

coveralls · 2025-11-19T21:01:32Z

Pull Request Test Coverage Report for Build 19674468060

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

For more information on this, see Tracking coverage changes with pull request builds.
To avoid this issue with future PRs, see these Recommended CI Configurations.
For a quick fix, rebase this PR at GitHub. Your next report should be accurate.

Details

0 of 0 changed or added relevant lines in 0 files are covered.
2 unchanged lines in 1 file lost coverage.
Overall coverage increased (+0.005%) to 96.414%

Files with Coverage Reduction	New Missed Lines	%
tests/test_resample.py	2	99.53%

Totals
Change from base Build 19515384246:	0.005%
Covered Lines:	56752
Relevant Lines:	58863

💛 - Coveralls

mraspaud

Thanks a lot for the work on supporting these HSAF products!
I especially appreciate the clarity of the code.

I have some comments, questions and suggestions inline, but I think this is almost ready :)

mraspaud · 2025-11-20T07:35:50Z

satpy/readers/hsaf_nc.py

+    """
+
+    def __init__(self, filename, filename_info, filetype_info):
+        """Opens the nc file and store the nc data."""


Suggested change

"""Opens the nc file and store the nc data."""

"""Open the nc file and store the nc data."""

mraspaud · 2025-11-20T07:50:16Z

satpy/readers/hsaf_nc.py

+        super().__init__(filename, filename_info, filetype_info)
+        self._area_name = None
+        self._lon_0 = None
+        # use generic_open contextmanager to handle compression transparently


Suggested change

# use generic_open contextmanager to handle compression transparently

mraspaud · 2025-11-20T07:50:27Z

satpy/readers/hsaf_nc.py

+        self._lon_0 = None
+        # use generic_open contextmanager to handle compression transparently
+        self._cm = generic_open(filename, mode="rb", compression="infer")
+        # manually enter the context manager


Suggested change

# manually enter the context manager

mraspaud · 2025-11-20T07:50:39Z

satpy/readers/hsaf_nc.py

+        self._cm = generic_open(filename, mode="rb", compression="infer")
+        # manually enter the context manager
+        fp = self._cm.__enter__()
+        # lazily load nc file through h5netcdf without forcing read


Suggested change

# lazily load nc file through h5netcdf without forcing read

mraspaud · 2025-11-20T07:53:56Z

satpy/readers/hsaf_nc.py

+    def __del__(self):
+        """Instruct the context manager to clean up and close the file."""
+        try:
+            if hasattr(self, "_cm") and self._cm:


Suggested change

if hasattr(self, "_cm") and self._cm:

I don't think this is needed: self._cm will we be present and have a value as long as the class has been instantiated. If not, this __del__ function will never be called anyway.

mraspaud · 2025-11-20T07:57:53Z

satpy/readers/hsaf_nc.py

+        self._area_name = self.filetype_info["area"]
+        self._lon_0 = float(self._nc_data.attrs["sub_satellite_longitude"].rstrip("f"))
+
+        # Add global attributes which are shared across datasets


Suggested change

# Add global attributes which are shared across datasets

mraspaud · 2025-11-20T07:58:48Z

satpy/readers/hsaf_nc.py

+    @property
+    def end_time(self):
+        """Get end time."""
+        return self.start_time + dt.timedelta(minutes=15)


15: So these products will never contain rss or fci data?

the products supported so far are all SEVIRI full-discs - when someone will add support for the FCI counterparts, this will need some refactoring

ok, how do we keep track of this?

looks like Alexandra made it configurable now, so it's solved I'd say

mraspaud · 2025-11-20T08:03:17Z

satpy/tests/reader_tests/test_hsaf_nc.py

+        with mock.patch("satpy.readers.hsaf_nc.generic_open") as mock_generic_open, \
+                mock.patch("satpy.readers.hsaf_nc.xr.open_dataset") as mock_open_dataset:


We have now as policy to try to create a stub file in tmp_path instead of mocking. I think here, it's a simple matter of writing the xarray dataset to disk and use that as input file. With the added benefit that you can (g/b)zip it to test the decompression...

mraspaud · 2025-11-20T08:06:22Z

satpy/tests/reader_tests/test_hsaf_nc.py

+    @pytest.mark.parametrize(
+        ("file_type", "loadable_ids"),
+        [
+            (FILE_PARAMS[FILE_TYPE_H60], ["rr", "qind"]),
+            (FILE_PARAMS[FILE_TYPE_H63], ["rr", "qind"]),
+            (FILE_PARAMS[FILE_TYPE_H90], ["acc_rr", "qind"]),
+        ],
+    )
+    @pytest.mark.skipif(not os.path.exists(FILE_PARAMS[FILE_TYPE_H60]["real_file"]) or
+                        not os.path.exists(FILE_PARAMS[FILE_TYPE_H63]["real_file"]),
+                                           reason="Real HSAF file not present")
+    def test_real_hsaf_file(self, file_type, loadable_ids):
+        """Test the reader with a real HSAF NetCDF file."""
+        # Select files
+        loadables = file_type["reader"].select_files_from_pathnames([file_type["real_file"]])
+        assert loadables, "No loadables found for the real file"


when are these real files available? are they easily downloadable from somewhere so that the user can test this?
Otherwise, I think it's as good to create stub files that mimic these files closely.

satpy/etc/enhancements/hsaf.yaml

This will make it possible to use this reader for other H-SAF products (e.g. H40/H40B) in the future, provided that the corresponding file type is configured.

...so that it complies with the guidelines for the use of stub files.

...so the defined enhancement will only be applied to `rr`.

armelzer · 2025-11-25T14:10:51Z

@ameraner @mraspaud The standard_name has been changed as suggested for rr and acc_rr, and I think we should also take into account all other comments/suggestions made so far. We have not made any specific improvements to acc_rr, as we are not experts in this field and in another PR .In addition, the delta value (in minutes) for the ‘end_time’ is now configurable across file types.

So thanks again for your exceptional support and feedback (also to @pnuu)!

With this commit the tear down method of the `TestHSAFNCReader` will be correctly called and the test for the enhancement will no longer fail.

albertbrotzer mentioned this pull request Nov 10, 2025

Add a reader for EUMETSAT HSAF SEVIRI H60 and H63 #3282

Open

ameraner requested changes Nov 10, 2025

View reviewed changes

ameraner assigned albertbrotzer Nov 10, 2025

ameraner added enhancement code enhancements, features, improvements component:readers labels Nov 10, 2025

ameraner added this to Remote PCW Autumn 2025 Nov 10, 2025

ameraner moved this to In Progress in Remote PCW Autumn 2025 Nov 10, 2025

ameraner reviewed Nov 10, 2025

View reviewed changes

satpy/tests/enhancement_tests/test_hsaf.py Outdated Show resolved Hide resolved

ameraner reviewed Nov 10, 2025

View reviewed changes

satpy/readers/hsaf_nc.py Outdated Show resolved Hide resolved

ameraner marked this pull request as ready for review November 12, 2025 09:23

ameraner requested review from djhoese and mraspaud as code owners November 12, 2025 09:23

ameraner requested changes Nov 12, 2025

View reviewed changes

albertbrotzer and others added 2 commits November 17, 2025 09:48

albertbrotzer force-pushed the feature-hsaf-msg branch from ab49c96 to f91dcc7 Compare November 17, 2025 09:48

albertbrotzer force-pushed the feature-hsaf-msg branch from f91dcc7 to 4e28107 Compare November 17, 2025 10:54

ameraner reviewed Nov 17, 2025

View reviewed changes

satpy/readers/hsaf_nc.py Outdated Show resolved Hide resolved

satpy/readers/hsaf_nc.py Outdated Show resolved Hide resolved

albertbrotzer added 2 commits November 18, 2025 14:54

Remove start/end time from the attributes

0c0b2ef

...as they're added automatically through the property methods

ameraner approved these changes Nov 19, 2025

View reviewed changes

Merge branch 'main' into feature-hsaf-msg

e514f54

pnuu approved these changes Nov 20, 2025

View reviewed changes

mraspaud requested changes Nov 20, 2025

View reviewed changes

armelzer added 4 commits November 21, 2025 10:52

Move definition of timedelta value for end_time calculation

bd2f94d

This will make it possible to use this reader for other H-SAF products (e.g. H40/H40B) in the future, provided that the corresponding file type is configured.

Clean up unnecessary comments and code

9778c0a

Create a stub files instead of mocking

41768cb

...so that it complies with the guidelines for the use of stub files.

Introduce separate standard_names for acc_rr and rr

7becf55

...so the defined enhancement will only be applied to `rr`.

Fix test execution

cba3797

With this commit the tear down method of the `TestHSAFNCReader` will be correctly called and the test for the enhancement will no longer fail.

albertbrotzer requested a review from mraspaud December 3, 2025 08:02

	"""Opens the nc file and store the nc data."""
	"""Open the nc file and store the nc data."""

		with mock.patch("satpy.readers.hsaf_nc.generic_open") as mock_generic_open, \
		mock.patch("satpy.readers.hsaf_nc.xr.open_dataset") as mock_open_dataset:

Add a reader for EUMETSAT HSAF SEVIRI H60 and H63 #3287

Are you sure you want to change the base?

Add a reader for EUMETSAT HSAF SEVIRI H60 and H63 #3287

Uh oh!

Conversation

albertbrotzer commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ameraner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ameraner commented Nov 10, 2025

Uh oh!

albertbrotzer commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Nov 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ameraner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ameraner commented Nov 12, 2025

Uh oh!

ameraner commented Nov 12, 2025

Uh oh!

albertbrotzer commented Nov 17, 2025

Uh oh!

ameraner left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ameraner left a comment

Choose a reason for hiding this comment

Uh oh!

pnuu commented Nov 19, 2025

Uh oh!

pnuu commented Nov 19, 2025

Uh oh!

albertbrotzer commented Nov 19, 2025

Uh oh!

pnuu commented Nov 19, 2025

Uh oh!

coveralls commented Nov 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Test Coverage Report for Build 19674468060

Warning: This coverage report may be inaccurate.

Details

💛 - Coveralls

Uh oh!

mraspaud left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

albertbrotzer commented Nov 10, 2025 •

edited

Loading

albertbrotzer commented Nov 12, 2025 •

edited

Loading

codecov bot commented Nov 12, 2025 •

edited

Loading

coveralls commented Nov 19, 2025 •

edited

Loading

armelzer commented Nov 25, 2025 •

edited

Loading