Nodata is not blending with the layer below

classic Classic list List threaded Threaded
12 messages Options
KingArthur10 KingArthur10
Reply | Threaded
Open this post in threaded view
|

Nodata is not blending with the layer below

I recently processed Texas NAIP data.  I did it as a 16-bit tif so that I could ensure I had a no-data value that wasn't in the 0-255 range and was lossless from the source material.

Anyway, in QGIS, the nodata value is perfectly utilized.  I can blend it with another image no problem.

qgis

When I then try to do the same thing in osgEarth, I get black borders around the entire state where nodata should be used and blended with the layer below.




I'm at a loss as to what's going on or why it's not blending, but this is critical for my needs.  Please advise.


My Texas gdalinfo is:

Driver: GTiff/GeoTIFF
Files: texas.tif
       texas.tif.ovr
       texas.tif.ovr.ovr
       texas.tif.aux.xml
Size is 1357416, 1108453
Coordinate System is:
GEOGCS["unnamed ellipse",
    DATUM["unknown",
        SPHEROID["unnamed",6378140,0]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (-106.694286338419715,36.586415781887304)
Pixel Size = (0.000009726297285,-0.000009726297285)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-106.6942863,  36.5864158) (106d41'39.43"W, 36d35'11.10"N)
Lower Left  (-106.6942863,  25.8052724) (106d41'39.43"W, 25d48'18.98"N)
Upper Right ( -93.4916548,  36.5864158) ( 93d29'29.96"W, 36d35'11.10"N)
Lower Right ( -93.4916548,  25.8052724) ( 93d29'29.96"W, 25d48'18.98"N)
Center      (-100.0929706,  31.1958441) (100d 5'34.69"W, 31d11'45.04"N)
Band 1 Block=256x256 Type=Int16, ColorInterp=Red
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 2 Block=256x256 Type=Int16, ColorInterp=Green
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 3 Block=256x256 Type=Int16, ColorInterp=Blue
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083


My basic .earth file is:
<map type="geocentric" version="2">
    
    <options>
        <terrain vertical_scale="3"/>
    </options>
    <image name="wholeearth" driver="gdal">
        <url>earth-imagery-bluemarble-16k.tif</url>
    </image>
    <!--Load a simple base image of the world-->
    <image name="world" driver="gdal">
        <url>texas.tif</url>
    </image>


    <options>
      <terrain driver="rex" tile_pixel_size="128" range_mode="PIXEL_SIZE_ON_SCREEN" skirt_ratio="0">
         <normal_maps>true</normal_maps>
         <lighting>true</lighting>
         <quick_release_gl_objects>false</quick_release_gl_objects>
         <cluster_culling>true</cluster_culling>
         <premultiplied_alpha>false</premultiplied_alpha>
         <vertical_scale>1</vertical_scale>
      </terrain>
    </options>
    
</map>

plevy plevy
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

I am trying to replicate the problem so can you run the osgEarth test and see if that works?

osgearth_viewer nodata.earth

Thanks
 

--
Paul Levy : Pelican Mapping


On Thu, Nov 15, 2018 at 7:07 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
I recently processed Texas NAIP data.  I did it as a 16-bit tif so that I could ensure I had a no-data value that wasn't in the 0-255 range and was lossless from the source material.

Anyway, in QGIS, the nodata value is perfectly utilized.  I can blend it with another image no problem.

qgis

When I then try to do the same thing in osgEarth, I get black borders around the entire state where nodata should be used and blended with the layer below.




I'm at a loss as to what's going on or why it's not blending, but this is critical for my needs.  Please advise.


My Texas gdalinfo is:

Driver: GTiff/GeoTIFF
Files: texas.tif
       texas.tif.ovr
       texas.tif.ovr.ovr
       texas.tif.aux.xml
Size is 1357416, 1108453
Coordinate System is:
GEOGCS["unnamed ellipse",
    DATUM["unknown",
        SPHEROID["unnamed",6378140,0]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (-106.694286338419715,36.586415781887304)
Pixel Size = (0.000009726297285,-0.000009726297285)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-106.6942863,  36.5864158) (106d41'39.43"W, 36d35'11.10"N)
Lower Left  (-106.6942863,  25.8052724) (106d41'39.43"W, 25d48'18.98"N)
Upper Right ( -93.4916548,  36.5864158) ( 93d29'29.96"W, 36d35'11.10"N)
Lower Right ( -93.4916548,  25.8052724) ( 93d29'29.96"W, 25d48'18.98"N)
Center      (-100.0929706,  31.1958441) (100d 5'34.69"W, 31d11'45.04"N)
Band 1 Block=256x256 Type=Int16, ColorInterp=Red
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 2 Block=256x256 Type=Int16, ColorInterp=Green
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 3 Block=256x256 Type=Int16, ColorInterp=Blue
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083


My basic .earth file is:
<map type="geocentric" version="2">
    
    <options>
        <terrain vertical_scale="3"/>
    </options>
    <image name="wholeearth" driver="gdal">
        <url>earth-imagery-bluemarble-16k.tif</url>
    </image>
    <!--Load a simple base image of the world-->
    <image name="world" driver="gdal">
        <url>texas.tif</url>
    </image>


    <options>
      <terrain driver="rex" tile_pixel_size="128" range_mode="PIXEL_SIZE_ON_SCREEN" skirt_ratio="0">
         <normal_maps>true</normal_maps>
         <lighting>true</lighting>
         <quick_release_gl_objects>false</quick_release_gl_objects>
         <cluster_culling>true</cluster_culling>
         <premultiplied_alpha>false</premultiplied_alpha>
         <vertical_scale>1</vertical_scale>
      </terrain>
    </options>
    
</map>




If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
plevy plevy
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

I grabbed a sample Texas NAIP image with a border and used :
gdal_translate -of GTiff -a_nodata 0 top08-nc-50cm_3403323_05082008.jp2 test.tif

and that works fine setting the nodata to black.

Do you have the steps you did for the 16bit version so I can try it here?
--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 12:25 PM plevy [via osgEarth] <[hidden email]> wrote:
I am trying to replicate the problem so can you run the osgEarth test and see if that works?

osgearth_viewer nodata.earth

Thanks
 

--
Paul Levy : Pelican Mapping


On Thu, Nov 15, 2018 at 7:07 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
I recently processed Texas NAIP data.  I did it as a 16-bit tif so that I could ensure I had a no-data value that wasn't in the 0-255 range and was lossless from the source material.

Anyway, in QGIS, the nodata value is perfectly utilized.  I can blend it with another image no problem.

qgis

When I then try to do the same thing in osgEarth, I get black borders around the entire state where nodata should be used and blended with the layer below.




I'm at a loss as to what's going on or why it's not blending, but this is critical for my needs.  Please advise.


My Texas gdalinfo is:

Driver: GTiff/GeoTIFF
Files: texas.tif
       texas.tif.ovr
       texas.tif.ovr.ovr
       texas.tif.aux.xml
Size is 1357416, 1108453
Coordinate System is:
GEOGCS["unnamed ellipse",
    DATUM["unknown",
        SPHEROID["unnamed",6378140,0]],
    PRIMEM["Greenwich",0],
    UNIT["degree",0.0174532925199433]]
Origin = (-106.694286338419715,36.586415781887304)
Pixel Size = (0.000009726297285,-0.000009726297285)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  COMPRESSION=LZW
  INTERLEAVE=PIXEL
Corner Coordinates:
Upper Left  (-106.6942863,  36.5864158) (106d41'39.43"W, 36d35'11.10"N)
Lower Left  (-106.6942863,  25.8052724) (106d41'39.43"W, 25d48'18.98"N)
Upper Right ( -93.4916548,  36.5864158) ( 93d29'29.96"W, 36d35'11.10"N)
Lower Right ( -93.4916548,  25.8052724) ( 93d29'29.96"W, 25d48'18.98"N)
Center      (-100.0929706,  31.1958441) (100d 5'34.69"W, 31d11'45.04"N)
Band 1 Block=256x256 Type=Int16, ColorInterp=Red
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 2 Block=256x256 Type=Int16, ColorInterp=Green
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083
Band 3 Block=256x256 Type=Int16, ColorInterp=Blue
  NoData Value=-32768
  Overviews: 169677x138557, 84839x69279, 42420x34640, 21210x17320, 10605x8660, 5303x4330, 2652x2165, 1326x1083


My basic .earth file is:
<map type="geocentric" version="2">
    
    <options>
        <terrain vertical_scale="3"/>
    </options>
    <image name="wholeearth" driver="gdal">
        <url>earth-imagery-bluemarble-16k.tif</url>
    </image>
    <!--Load a simple base image of the world-->
    <image name="world" driver="gdal">
        <url>texas.tif</url>
    </image>


    <options>
      <terrain driver="rex" tile_pixel_size="128" range_mode="PIXEL_SIZE_ON_SCREEN" skirt_ratio="0">
         <normal_maps>true</normal_maps>
         <lighting>true</lighting>
         <quick_release_gl_objects>false</quick_release_gl_objects>
         <cluster_culling>true</cluster_culling>
         <premultiplied_alpha>false</premultiplied_alpha>
         <vertical_scale>1</vertical_scale>
      </terrain>
    </options>
    
</map>




If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML



If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592161.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
KingArthur10 KingArthur10
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

Let me see what I can whip up as a small sample test case.  Sorry, sending you a 2TB file isn't easy ;).

The nodata test file works fine, so I think it has something to do with the Int16 format, so I'm seeing what I can do to replicate it.
KingArthur10 KingArthur10
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

In reply to this post by KingArthur10
Try this file:

https://drive.google.com/open?id=1lznB-ZdQY3mja9NAg4riVphvToS6XdLe

I simply took the nodata file that was used, and converted it to 16-bit and stripped it to 3-bands rather than 4.

gdalwarp -ot Int16 -dstnodata -32768 nodata.tif nodata16.tif

gdal_translate -ot Int16 -b 1 -b 2 -b 3 nodata16.tif nodata16-1.tif

Reproduces the error.

Let me know how else I can help.
plevy plevy
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

OK thanks. Are you planning to leave the data in 16bit?  Can you generate the alpha band from the 16bit and use it on the original 8bit to remove the proper black parts?

--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 2:21 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
Try this file:

https://drive.google.com/open?id=1lznB-ZdQY3mja9NAg4riVphvToS6XdLe

I simply took the nodata file that was used, and converted it to 16-bit and stripped it to 3-bands rather than 4.

gdalwarp -ot Int16 -dstnodata -32768 nodata.tif nodata16.tif

gdal_translate -ot Int16 -b 1 -b 2 -b 3 nodata16.tif nodata16-1.tif

Reproduces the error.

Let me know how else I can help.


If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592165.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
plevy plevy
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

I see it here with the test file and I am thinking along the same lines, something with the 16bit format.

I'll take a look

Thanks
--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 3:42 PM plevy [via osgEarth] <[hidden email]> wrote:
OK thanks. Are you planning to leave the data in 16bit?  Can you generate the alpha band from the 16bit and use it on the original 8bit to remove the proper black parts?

--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 2:21 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
Try this file:

https://drive.google.com/open?id=1lznB-ZdQY3mja9NAg4riVphvToS6XdLe

I simply took the nodata file that was used, and converted it to 16-bit and stripped it to 3-bands rather than 4.

gdalwarp -ot Int16 -dstnodata -32768 nodata.tif nodata16.tif

gdal_translate -ot Int16 -b 1 -b 2 -b 3 nodata16.tif nodata16-1.tif

Reproduces the error.

Let me know how else I can help.


If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592165.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML



If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592166.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
plevy plevy
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

It doesn't look like 16bit is going to work right now as it assumes 8bits and then checks against -32767, fails and draws the color was black.


--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 4:04 PM plevy [via osgEarth] <[hidden email]> wrote:
I see it here with the test file and I am thinking along the same lines, something with the 16bit format.

I'll take a look

Thanks
--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 3:42 PM plevy [via osgEarth] <[hidden email]> wrote:
OK thanks. Are you planning to leave the data in 16bit?  Can you generate the alpha band from the 16bit and use it on the original 8bit to remove the proper black parts?

--
Paul Levy : Pelican Mapping


On Fri, Nov 16, 2018 at 2:21 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
Try this file:

https://drive.google.com/open?id=1lznB-ZdQY3mja9NAg4riVphvToS6XdLe

I simply took the nodata file that was used, and converted it to 16-bit and stripped it to 3-bands rather than 4.

gdalwarp -ot Int16 -dstnodata -32768 nodata.tif nodata16.tif

gdal_translate -ot Int16 -b 1 -b 2 -b 3 nodata16.tif nodata16-1.tif

Reproduces the error.

Let me know how else I can help.


If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592165.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML



If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592166.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML



If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592167.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
KingArthur10 KingArthur10
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

I guess I'll rescale the values, but that seems like osgEarth shouldn't be bound just to byte-type data.  Makes it impossible to have black in an image.
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

Not sure what you mean, black is just RGB 0,0,0.  Normally with RGB data that is 16 bit the values have to be scaled via a histogram range to look sensible anyway, and we usually just defer that to a GDAL VRT with a scaling function anyway.  

On Fri, Nov 16, 2018 at 8:22 PM KingArthur10 [via osgEarth] <[hidden email]> wrote:
I guess I'll rescale the values, but that seems like osgEarth shouldn't be bound just to byte-type data.  Makes it impossible to have black in an image.


If you reply to this email, your message will be added to the discussion below:
http://forum.osgearth.org/Nodata-is-not-blending-with-the-layer-below-tp7592158p7592169.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.
NAML
ADB ADB
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

If your no data value is equivalent to Black, then black becomes transparent and blends with other layers. The problem is, other layers may not have black in that spot. For example, if the time of the year is different, black in a high res layer May corresponds to Green in a lower-res layer.

What I have done is scaled all the values from 0 to 255 into the range of 1 to 255. That way, I can set the no data value to zero and not risk black being confused with no data. But, it is not the ideal way to do it as far as I can tell.

0,0,0 does not imply no data.

The initial way that I went about it is to take my byte format data as the input through the VRT files and translate that into a 16-bit tiff preserving the 0 to 255 values that were in the original data set and setting the no data value to be a negative number. That way, I didn't actually have to alter the source data at all and places that did not have data would accurately be represented by a no data value.

Does it sound like I am doing something wrong? Do you have a better way to do it?

Arthur
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Nodata is not blending with the layer below

Hey Arthur,

No data doesn't have to be equivalent to black.  It's appearing as black in osgearth b/c osgearth is assuming that imagery datasets have already been pre-scaled to byte range (0-255). Your int16 data has a nodata of -32768 and contains pixels that are -32768.  When osgearth reads one of these pixels, it is assuming that it is an unsigned char, so it ends up being clamped to 0 because -32768 is outside of that range.  When 0 is then compared to your nodata value of -32768 osgearth thinks that 0 is a valid value, so it just passes it through.

You say:
"What I have done is scaled all the values from 0 to 255 into the range of 1 to 255. That way, I can set the no data value to zero and not risk black being confused with no data. But, it is not the ideal way to do it as far as I can tell."

You're basically just running into a situation that osgearth doesn't handle and is out of the ordinary.  Typically with imagery it should just be scaled to byte range and if you have "nodata" areas you'll provide an alpha band with the values of 1 where you have data and 0 where you don't.  

If the values in your dataset are actually scaled between 0 and 255 you're actually making a much bigger file than you need by providing 3 int16 bands and a negative nodata rather than 4 byte bands (RGBA).

So I'd recommend just switching to a byte dataset and creating an alpha band to represent your nodata areas instead and you should be be good to go.

Jason