memory/ processor usage

classic Classic list List threaded Threaded
24 messages Options
12
tkani tkani
Reply | Threaded
Open this post in threaded view
|

memory/ processor usage

Hi , the processor and especially the memory usage seems to have problem on my project.
Here is a sample :
--------------------------------------------------
<map name="srtm sample" type="globe">
       
        <cache> 
          <path>osgearth_cache</path> 
        </cache>
        <image name="world" driver="gdal">
                <url>data/world.tif</url>
                <cache_only>true</cache_only>
                <cache_format>tif</cache_format>
        </image>
</map>
--------------------------------------------------

Here the graph of the memory usage, at the first step the osgviewer is not launch, then I launch it (we can see the processor usage being more important and the memory getting a little bit higher (80MB)) and then I do a zoom on a location wherever in the world, this is the last part of the graph , a very steep slope going up....

any suggestions ?  thanks !

Thomas canipel
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

I forgot to precise that I actually work with the latest version of osgearth.
After some test it appears that ogearth re-allocate  always the lock.
Thomas canipel
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi,

I'm looking into this right now, I can reproduce it.

Jason

On Tue, Feb 16, 2010 at 7:44 PM, tcanipel [via osgEarth]
<[hidden email]> wrote:

> I forgot to precise that I actually work with the latest version of
> osgearth.
> After some test it appears that ogearth re-allocate  always the lock.
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4583595.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi again,

Try the trunk, the issue should be fixed there.  The issue was that
tiles that were never being added to the scene graph were registering
tiles with VersionedTerrain object.  I've changed the logic of when
tiles are registered so that they are only registered when they are
first traversed, and therefore added to the scene graph.  This fixes
the memory leak for me.

Thanks,

Jason

On Tue, Feb 16, 2010 at 7:45 PM, Jason Beverage <[hidden email]> wrote:

> Hi,
>
> I'm looking into this right now, I can reproduce it.
>
> Jason
>
> On Tue, Feb 16, 2010 at 7:44 PM, tcanipel [via osgEarth]
> <[hidden email]> wrote:
>> I forgot to precise that I actually work with the latest version of
>> osgearth.
>> After some test it appears that ogearth re-allocate  always the lock.
>>
>> ________________________________
>> View message @
>> http://n2.nabble.com/memory-processor-usage-tp4583418p4583595.html
>> To start a new topic under osgEarth, email
>> [hidden email]
>> To unsubscribe from osgEarth, click here.
>>
>
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

yes the new version seems to resolve the big memory leak but I am still sure that some tiles when you zoom +/- are not desallocate correctly, try to zoom and unzoom on the world.tif, the memory usage will still grow up a little bit.., and on a long time it seems to use more and more memory.
Thomas canipel
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

What type of memory levels are you seeing?  It looks very stable to
me, zooming and out the numbers look reasonable.

Jason

On Tue, Feb 16, 2010 at 8:59 PM, tcanipel [via osgEarth]
<[hidden email]> wrote:

> yes the new version seems to resolve the big memory leak but I am still sure
> that some tiles when you zoom +/- are not desallocate correctly, try to zoom
> and unzoom on the world.tif, the memory usage will still grow up a little
> bit.., and on a long time it seems to use more and more memory.
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4583813.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

for example by launching the visualization of world.tif I can easily go to 150MB usage, when for example you zoom on a special location (the memory usage increase) , and then zoom out to see the entire earth for a while, I was expecting that the memory usage decrease after a while.
Thomas canipel
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

hi ! I have done more testand there is another memory leak, in order to see it it will be good to have the same sample to work on:
 this link download the data for the heifield of the lake tahoe:
http://tahoe.usgs.gov/products/dem/dembathy_dem.zip

here is the sample .earth:
---------------------------------------------
<map name="srtm sample" type="globe">
        <image name="world" driver="gdal">
                <url>data/world.tif</url>
                <cache_format>tif</cache_format>
        </image>
        <heightfield name="tahoe" driver="gdal">
                <url>data/dembathyp.dem</url>
                <tile_size>64</tile_size>
        </heightfield>
</map>
-------------------------------------------------------------------------

it seems that the reprojection still have a problem for this file so just use gdalwarp to reproject it :

gdalwarp -t_srs epsg:4326 dembathy.dem dembathyp.dem


and now you can zoom in zoom out on the lake tahoe (california), go to another continent , zoom out and then go to california to zoom in on the lake, the memory will always increase, one thing I dont understand either is that the LOD seems to be loaded when you zoom in, then you zoom out the LOD is still in the memory but when you zoom in again the engine reload the LODs and increase the memory instead of switching with the LOD he already has.

Thanks !
Thomas canipel
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi,

I've committed a fix for the heightfield reprojection issue in the trunk.

Also, I'm not seeing a memory leak at all when using either the warped
or reprojected files.  It's normal for the memory not to completely go
down to it's beginning state when you zoom all the way out b/c tiles
are expired by the osg DatabasePager when a max limit is reached.  If
I zoom in on tahoe, my memory increases, then when I zoom out to the
world view, memory stays about the same.  When I zoom into another
continent, memory drops b/c the tahoe tiles are expired.  That is why
when you zoom into tahoe again, it has to reload the tiles.  You can
use a cache to speed this up, or increase your OSG_MAX_PAGEDLOD
environment variable to keep more tiles in memory (default is 300).

Also, what is your first name so we know how to address you on the list?

Thanks!

Jason

On Tue, Feb 16, 2010 at 10:10 PM, tcanipel [via osgEarth]
<[hidden email]> wrote:

> hi ! I have done more testand there is another memory leak, in order to see
> it it will be good to have the same sample to work on:
>  this link download the data for the heifield of the lake tahoe:
> http://tahoe.usgs.gov/products/dem/dembathy_dem.zip
>
> here is the sample .earth:
> ---------------------------------------------
> <map name="srtm sample" type="globe">
>         <image name="world" driver="gdal">
>                 <url>data/world.tif</url>
>                 <cache_format>tif</cache_format>
>         </image>
>         <heightfield name="tahoe" driver="gdal">
>                 <url>data/dembathyp.dem</url>
>                 <tile_size>64</tile_size>
>         </heightfield>
> </map>
> -------------------------------------------------------------------------
>
> it seems that the reprojection still have a problem for this file so just
> use gdalwarp to reproject it :
>
> gdalwarp -t_srs epsg:4326 dembathy.dem dembathyp.dem
>
>
> and now you can zoom in zoom out on the lake tahoe (california), go to
> another continent , zoom out and then go to california to zoom in on the
> lake, the memory will always increase, one thing I dont understand either is
> that the LOD seems to be loaded when you zoom in, then you zoom out the LOD
> is still in the memory but when you zoom in again the engine reload the LODs
> and increase the memory instead of switching with the LOD he already has.
>
> Thanks !
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4584027.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi, here is the description of the memory leak its not as important as the last one but still present :

(as I have problem to upload the image on the forum right now (header pb) here is the link to see the image on my picasa account )

http://picasaweb.google.com/thomas.canipel/MemoryLeak#

hope it will help you to see the memory leak, btw i notice something weird about the LOD loading, when you zoom on an area where there is nothing (for example africa) the number of request is pretty impressive for the number of vertices there is ....

thanks!




Thomas canipel
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi jason have ou success to reproduce the leak ?

thanks
Thomas canipel
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi Thomas,

Do me a quick favor and send me an animation path that reproduces the
problem for you.  Just start osgviewer, hit z to start recording, zoom
around until the problem presents itself obviously and then hit
shift-z to write out the animation path.  That will help me in
debugging.

Also, are you seeing this issue with your reprojected latlong version
of the tahoe dataset or with the original UTM version.

Thanks,

Jason

On Thu, Feb 18, 2010 at 1:28 PM, tcanipel [via osgEarth]
<[hidden email]> wrote:

> Hi jason have ou success to reproduce the leak ?
>
> thanks
> Thomas canipel
>
> Thanks
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4593433.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi jason here is the link of the animation :

http://uploading.com/files/fb498b41/saved_animation.path/

I use the file already reprojected with gdalwarp like taht I was sure that it didnt come from the reprojection update.

thanks


Thomas canipel
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

BTW you have tried with the file non reprojected, i got the error message :

[osgEarth::SRS] Failed to xform a point from WGS84 to UTM Zone 10, Northern Hemisphere

this is a plugin he couldnt load to do the projection or there is still a problem in the reprojection inside osgearth?

thanks
Thomas canipel
tkani tkani
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Have you play with the tahoe for like 5min going everywhere in the world  to see that your memory increase a lot until it crash ?

sorry to ask you a lot about that but the memory leak is pretty obvious when you play with the earth, so perhaps it come from the data (perhaps I have to tiled it or do more operations on it) or by setting the cache correctly or another solution to slow down the increase of the memory usage...

thanks !
Thomas canipel
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi Thomas,

I'm looking at the issue right now, patience :)

I'll let you know what I find.

Thanks,

Jason

On Fri, Feb 19, 2010 at 2:36 PM, tcanipel [via osgEarth]
<[hidden email]> wrote:

> Have you play with the tahoe for like 5min going everywhere in the world  to
> see that your memory increase a lot until it crash ?
>
> sorry to ask you a lot about that but the memory leak is pretty obvious when
> you play with the earth, so perhaps it come from the data (perhaps I have to
> tiled it or do more operations on it) or by setting the cache correctly or
> another solution to slow down the increase of the memory usage...
>
> thanks !
> Thomas canipel
>
> Thanks
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4599691.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

In reply to this post by tkani
Hi Thomas,

After much debugging and a few new gray hairs, I've committed a fix
for this memory leak.  The issue was introduced in r878 when the list
of Terrain's was added to it's own group.  What was happening was that
the OSG optimizer was determining that the new _terrains group wasn't
necessary and optimizing it away by adding the Terrains directly to
the MapNode.  However, because we kept a reference to the _terrains
node, the Terrain objects now had two parents: the MapNode and the
original _terrains Group, which was no longer part of the scene graph.

Because the Terrain now had two parents, the findCam function that
attempted to traverse the Terrain's parents to find the root Camera
object wasn't able to find it because it assumed that the Terrain
would only have one parent.  Therefore, when a node expired, it was
added to the _tilesToRelease list, which was never cleared because the
callback could not be installed.  And thus, memory leaked all over the
place :)

My fix protects the _terrains group from being optimized away by
giving it a StateSet, removed the findCam method and replaced with a
call to the new findFirstParentOfType function, which should take into
account multiple parents (although this shouldn't happen anymore, just
being safe).  I've also made sure to only add tiles to the
_tilesToRelease list if we could actually install the callback,
otherwise I'm printing a warning.

Memory is much more stable now, try the trunk and let me know if it
works better for you.

Thanks!

Jason

On Fri, Feb 19, 2010 at 2:40 PM, Jason Beverage <[hidden email]> wrote:

> Hi Thomas,
>
> I'm looking at the issue right now, patience :)
>
> I'll let you know what I find.
>
> Thanks,
>
> Jason
>
> On Fri, Feb 19, 2010 at 2:36 PM, tcanipel [via osgEarth]
> <[hidden email]> wrote:
>> Have you play with the tahoe for like 5min going everywhere in the world  to
>> see that your memory increase a lot until it crash ?
>>
>> sorry to ask you a lot about that but the memory leak is pretty obvious when
>> you play with the earth, so perhaps it come from the data (perhaps I have to
>> tiled it or do more operations on it) or by setting the cache correctly or
>> another solution to slow down the increase of the memory usage...
>>
>> thanks !
>> Thomas canipel
>>
>> Thanks
>>
>> ________________________________
>> View message @
>> http://n2.nabble.com/memory-processor-usage-tp4583418p4599691.html
>> To start a new topic under osgEarth, email
>> [hidden email]
>> To unsubscribe from osgEarth, click here.
>>
>
Glenn Waldron Glenn Waldron
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Good God, nice find Jason. The optimizer strikes again.

Glenn
rschmidt rschmidt
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

In reply to this post by jasonbeverage
Hi,
somehow the memory is still growing for me... and I am using revision 901.

It does not seem to be a driver related problem. The memory increases with all drivers.

Richard

PS.: Is there something like osgEarth-submissions?
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: memory/ processor usage

Hi,

Are you using osgviewer or your own application?  With the fix I just
committed, I noticed the memory increase in all drivers as well and it
seemed to be stabilized with the fix.

No, there isn't a separate osgEarth-submissions, you can send any
submissions to this list.

Jason

On Mon, Feb 22, 2010 at 7:36 AM, rschmidt [via osgEarth]
<[hidden email]> wrote:

> Hi,
> somehow the memory is still growing for me... and I am using revision 901.
>
> It does not seem to be a driver related problem. The memory increases with
> all drivers.
>
> Richard
>
> PS.: Is there something like osgEarth-submissions?
>
> ________________________________
> View message @
> http://n2.nabble.com/memory-processor-usage-tp4583418p4611754.html
> To start a new topic under osgEarth, email
> [hidden email]
> To unsubscribe from osgEarth, click here.
>
12