Nikon D5300 – Lossy NEF Compression (AKA Nikon Concentric ring problem)

So, one of the huge benefits to shooting RAW on a DSLR is the increase in image quality over JPEG – you are recording exactly what the sensor sees, as compared to a “visually lossless” image such as a JPEG.

Of course, JPEG was mostly designed around regular images of the world around us and not extremely dark images such as our astrophotography sub-frames.
Also, as we stretch and manipulate our images, the JPEG data can easily break down, exposing compression artifacts and flaws.

So enter RAW, the perfect solution, right ? Well, maybe. There are a number of cameras that perform a number of manipulations to RAW files. Far too many to go into all of them here (and I am far from an expert).

Some can be turned off (Long-Exposure Noise Reduction – where the camera essentially takes a dark for every frame and subtracts it before giving you the RAW data).

Some can only be turned off by modding the firmware (i.e. the Nikon D5100 can be modified with custom firmware that enables a “pedastal” to ensure that dark frames and dark subs do not get clipped – cameras like the Nikon D5300 do this automatically)

However, there are some cameras (such as the Nikon D5300) who’s firmware doesn’t (and probably will never) have a modded version. Some of these cameras do very odd things to RAW data under certain circumstances, and sometimes the best we can do is to try to avoid it or develop software to correct it.

The Nikon D5300 (and a number of other models – see below) actually performs “visually lossless” compression on both 12 and 14-bit RAW frames and there is currently no way to turn it off. This is far far better than JPEG compression, but can still be an issue under certain cases (one such is theorized to be very dark exposures — exactly like our astrophotography sub-frames).

Various theories have been extended

expose sub-frames more and flat frames less
The moon is part of the cause
It doesn’t happen as much at a dark site
It happens less with less vignette (so using a refractor with a wider image circle may help)

Non of these are foolproof, and so sometimes mitigation is the only remedy.

A very enterprising Cloudy Nights member has led the charge and done extensive analysis and has discovered that the compression results in histogram “stairstepping” in the Red and Blue channels but the green is intact (lossless compression). This led him to implement software to correct the issue.

The correction is not perfect but offers a vast improvement over the default compression. The software works on DNG files, so you must download and install Adobe DNG converter (free)

The process is then very simple

Use Adobe converter to convert flats & light frames to DNG (I also did darks but I do not think this is necessary)
Run the windows program (see below) to process the DNGs and remove the rings
Calibrate and integrate the processed DNG files

Here is the link to Cloudy Nights with extensive research and details

Here is the link on Cloudy Nights to the member (Mark) who did the bulk of the work to correct the images and develop the correction software

Adobe DNG Converter: https://helpx.adobe.com/photoshop/using/adobe-dng-converter.html

The correction software: (V3 is the current): https://drive.google.com/file/d/14IHauD0sMecsjrOCTZ8TBatLUO_Xs17a

List of cameras affected: https://www.cloudynights.com/topic/746131-nikon-coloured-concentric-rings/?p=10867894

Here is an example of a before and after on one of my images. Quite an astounding improvement