Long time of layers' switching

classic Classic list List threaded Threaded
23 messages Options
12
slawowski slawowski
Reply | Threaded
Open this post in threaded view
|

Long time of layers' switching

Hi,

I have updated to osgEarth 1.2 and now I'm using it's new API to manipulate layers on the earth programmatically. Generally the new API structure is well organized, but I'm facing the following problem - when I come closer to the earth and switch the layers, my program hangs (I cannot manipulate the earth) and it takes a long time before the new layer appears and I can move the earth again. The break is very long for Internet sources (google) - over one minute. For local data it is also significant.

It seems that osgEarth waits until tiles for all LODs up to the currently visible are downloaded - am I right? Is there any way reduce this effect? I'd prefer a behavior similar to normal earth viewing - loading more accurate tiles during earth manipulating, without any breaks. I achieved such a behavior in osgEarth 1.1, when I was creating new Nodes every time I was switching layers in my program.

Here is a code snippet responsible for switching layers in my program when using osgEarth 1.2. I hope I'm doing it correctly:

    mapNode->getMap()->removeMapLayer(vpbLayer);
    Properties conf;
    conf["dataset"] = "labels";
    googleLayer = new MapLayer("image_google", MapLayer::TYPE_IMAGE, "google", conf);
    mapNode->getMap()->addMapLayer(googleLayer);


Best regards,
Slawek
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi Slawek,

You are correct, currently osgEarth loads all the data from the new source for the existing tiles, so it can cause a stall.  We are currently investigating new ways to speed up the performance of adding and removing layers with the layer API, so keep an eye out!

Thanks!

Jason

On Fri, Aug 14, 2009 at 9:30 AM, slawowski (via Nabble) <[hidden email]> wrote:
Hi,

I have updated to osgEarth 1.2 and now I'm using it's new API to manipulate layers on the earth programmatically. Generally the new API structure is well organized, but I'm facing the following problem - when I come closer to the earth and switch the layers, my program hangs (I cannot manipulate the earth) and it takes a long time before the new layer appears and I can move the earth again. The break is very long for Internet sources (google) - over one minute. For local data it is also significant.

It seems that osgEarth waits until tiles for all LODs up to the currently visible are downloaded - am I right? Is there any way reduce this effect? I'd prefer a behavior similar to normal earth viewing - loading more accurate tiles during earth manipulating, without any breaks. I achieved such a behavior in osgEarth 1.1, when I was creating new Nodes every time I was switching layers in my program.

Here is a code snippet responsible for switching layers in my program when using osgEarth 1.2. I hope I'm doing it correctly:

    mapNode->getMap()->removeMapLayer(vpbLayer);
    Properties conf;
    conf["dataset"] = "labels";
    googleLayer = new MapLayer("image_google", MapLayer::TYPE_IMAGE, "google", conf);
    mapNode->getMap()->addMapLayer(googleLayer);


Best regards,
Slawek


View message @ http://n2.nabble.com/Long-time-of-layers%27-switching-tp3445055p3445055.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


slawowski slawowski
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi,

Thanks for your reply. Today I hit on the following idea: what if we remove all existing tiles when adding or removing layers? This way we would force the engine to reload the tiles with new layers and we will see how the tiles are loaded, so there will be no stall.

Is my idea correct? If yes, how can I remove all existing tiles from MapNode?

Best regards,
Slawek
gwaldron gwaldron
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Actually, the fastest thing would be to remove the MapNode itself and just replace it with a new MapNode with the new layer set.

BTW-- for the next version of osgearth, one of the main items will be a new layer management implementation that will resolve this issue. We wanted to get a working API out there for 1.2, and then move towards making it performant in 1.3

Glenn
slawowski slawowski
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi,

Now I'm using this method which removes the MapNode and replaces it with a new one with different layer set. Unfortunately, after executing this method for a few times, PagedLOD children do not load anymore. I can see only the least accurate LOD, even when I come closer to the earth. When this situation occurs once, every further change also causes the same problem - LODs do not load.

I checked that OSG database pager accepts the requests but does not return any scene graphs, also for requests not connected with the earth.

Now I remove the MapNode by simple removing it from the parents' group. Is there any other special way I should do it? I tried removing all layers form MapNode's Map before detaching it but it didn't help.

Thanks,
Slawek
gwaldron gwaldron
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi Slawek,

From what you describe, it almost sounds like the database pager itself if dying. You say it stops returning scene graphs, even from non-osgEarth PagedLODs? We'll take a look and try to duplicate the problem.

In the meantime, you might want to try the new mechanism in SVN called "deferred loading". When enabled, two things happen:

1) The highest LOD available for your current camera position loads first (rather than having to go through all the lower LODs); and

2) Layer add/remove/reorder is instantaneous.

This is still a work in progress but so far works well. See the "arc_deferred.earth" sample; or you can run "osgearth_toc --deferred" to try it out. From code you would call MapEngineProperties.setDeferTileDataLoading(true) before creating your MapNode.

Like I said it is still a work in progress... for instance it does not yet work properly with spherical mercator datasources like OpenStreetMap.

Glenn
Pelican Mapping
Glenn Waldron / Pelican Mapping
slawowski slawowski
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi,

I am posting in this old thread, because I would like to ask about current performance of the three available loading policies: standard, sequential and preemptive. As I described earlier, I want to change my layer set during program execution. So far I was replacing my earth root with a new MapNode with new layer set every time the change occurred. Generally, this solution is quite efficient and looks acceptably. However, it causes some problems in my application, because it replaces pointers to nodes in the scene graph, which are used by me in other structures in my application.

Therefore, when osgEarth 1.3 is released, I want to change the layer set using the API provided in osgEarth. The most efficient for me is preemptive mode, because it does not cause a stall after the layers set has changed. Unfortunately, the visual effects in this policy are not fully satisfiable for me.

I also tested standard and sequential mode, but it seems that they are not free of the problem I mentioned at the beginning of this thread: my application hangs for quite a long time each time I change the layers set.

Could you please tell me whether my observations are correct? Do you suggest any solution in my situation?

Best regards,
Slawek
JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by gwaldron
Glenn,

I realize this is a fairly old thread, but were you guys able to reproduce Slawek's problem of the database pager dying? I'm seeing the same thing here. And you are correct in assuming that even non-osgEarth PagedLODs do not load.

Perhaps it will be a question for the osg forum, but since this topic is here, I thought I'd check with you guys first.

Thanks,
Jesse

gwaldron wrote
Hi Slawek,

From what you describe, it almost sounds like the database pager itself if dying. You say it stops returning scene graphs, even from non-osgEarth PagedLODs? We'll take a look and try to duplicate the problem.

In the meantime, you might want to try the new mechanism in SVN called "deferred loading". When enabled, two things happen:

1) The highest LOD available for your current camera position loads first (rather than having to go through all the lower LODs); and

2) Layer add/remove/reorder is instantaneous.

This is still a work in progress but so far works well. See the "arc_deferred.earth" sample; or you can run "osgearth_toc --deferred" to try it out. From code you would call MapEngineProperties.setDeferTileDataLoading(true) before creating your MapNode.

Like I said it is still a work in progress... for instance it does not yet work properly with spherical mercator datasources like OpenStreetMap.

Glenn
Pelican Mapping
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi Jesse,

Under what circumstances is the pager dying?  Are you just viewing the osgEarth map and moving around or are you adding/removing layers, etc.

Thanks,

Jason

On Tue, Jan 12, 2010 at 10:06 AM, JesseStimpson [via osgEarth] <[hidden email]> wrote:
Glenn,

I realize this is a fairly old thread, but were you guys able to reproduce Slawek's problem of the database pager dying? I'm seeing the same thing here. And you are correct in assuming that even non-osgEarth PagedLODs do not load.

Perhaps it will be a question for the osg forum, but since this topic is here, I thought I'd check with you guys first.

Thanks,
Jesse

gwaldron wrote:
Hi Slawek,

From what you describe, it almost sounds like the database pager itself if dying. You say it stops returning scene graphs, even from non-osgEarth PagedLODs? We'll take a look and try to duplicate the problem.

In the meantime, you might want to try the new mechanism in SVN called "deferred loading". When enabled, two things happen:

1) The highest LOD available for your current camera position loads first (rather than having to go through all the lower LODs); and

2) Layer add/remove/reorder is instantaneous.

This is still a work in progress but so far works well. See the "arc_deferred.earth" sample; or you can run "osgearth_toc --deferred" to try it out. From code you would call MapEngineProperties.setDeferTileDataLoading(true) before creating your MapNode.

Like I said it is still a work in progress... for instance it does not yet work properly with spherical mercator datasources like OpenStreetMap.

Glenn
Pelican Mapping



View message @ http://n2.nabble.com/Long-time-of-layers-switching-tp3445055p4292112.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by slawowski
Hi Slawek,

In "standard" mode, osgEarth expects all of the tiles in the scene graph to contain all of the imagery and elevation data as soon as it's created by the DatabasePager.  So, when you add a layer in standard mode, all of the existing TerrainTiles are updated to reflect the new imagery, which is why it takes so long in standard mode.

In "sequential" and "preemptive" mode, the data can more easily be replaced from a background thread because it is using a different thread pool other than the DatabasePager and the TerrainTile's contain placeholders that can be replaced later.  Preemptive mode just adds the imagery data as soon as it comes in, no matter what LOD it is at, so higher resolution imagery might come in before lower resolution imagery, causing a popping effect, which I assume is what you are talking about when you say the "visual effects in this policy are not satisfiable for me".  In sequential mode, the imagery is loaded using our own thread pool, but it attempts to load the LOD's in order so that you don't see popping effects and should look more like the standard mode does.  The first level of detail is always immediately loaded in sequential mode so that higher LOD's can progress.

I just did some tests with running osgearth_toc --sequential and the time it takes to add a new layer is not unreasonable to me, around 5 or 6 seconds before you can start to move the scene again.    What kind of times are you seeing?  The adding and removing of layers isn't currently implemented to be instantaneous.

Thanks,

Jason

On Mon, Jan 11, 2010 at 4:50 PM, slawowski [via osgEarth] <[hidden email]> wrote:
Hi,

I am posting in this old thread, because I would like to ask about current performance of the three available loading policies: standard, sequential and preemptive. As I described earlier, I want to change my layer set during program execution. So far I was replacing my earth root with a new MapNode with new layer set every time the change occurred. Generally, this solution is quite efficient and looks acceptably. However, it causes some problems in my application, because it replaces pointers to nodes in the scene graph, which are used by me in other structures in my application.

Therefore, when osgEarth 1.3 is released, I want to change the layer set using the API provided in osgEarth. The most efficient for me is preemptive mode, because it does not cause a stall after the layers set has changed. Unfortunately, the visual effects in this policy are not fully satisfiable for me.

I also tested standard and sequential mode, but it seems that they are not free of the problem I mentioned at the beginning of this thread: my application hangs for quite a long time each time I change the layers set.

Could you please tell me whether my observations are correct? Do you suggest any solution in my situation?

Best regards,
Slawek


View message @ http://n2.nabble.com/Long-time-of-layers-switching-tp3445055p4288377.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by jasonbeverage
Jason,

I've been able to reproduce by recreating a new MapNode while osgEarth was paging in new terrain geometry. However, I can't get it down to a consistent use case, unfortunately.

I believe that in all cases where I've seen the paging stop, I was doing something intrusive with the Map (recreating it, moving layers, etc).

I'll let you know if I come across something more concrete.

Thanks,
Jesse
slawowski slawowski
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by jasonbeverage
Hi Jason,

Thanks for your clarifications. I have tested "sequential" mode once again and now I see a positive difference between this one and "standard" mode. Unfortunately, my program still hangs for more than just few seconds. Here are the details.

When I change the layer set, I remove all the existing layers and add new ones, even if they contain layers which were already present in the map (I know this could be optimized, but there may be cases in which all the layers are replaced). Initially I have only one raster layer from my local VPB database, so it loads quickly all necessary tiles even when I am close to earth. Then I add an online layer, for example OpenStreetMap. The program hangs for a second or two so it is acceptable. Then, I remove the online layer - I do it when I am close to earth and tiles from the online layer were still loading. This is the situation when the problem occurs - my program hangs for about 1-3 minutes, which is not a reasonable time.

Then I repeated this scenario, but looking at the earth from a much higher point, so that I could see the whole globe. In this case both adding and removing layers worked fast, the program did not hang for more than 1 second.

To sum up, it seems that in "sequential" mode especially time consuming is removing a layer which requested many tiles (because of view point set close to earth terrain). Do you agree with my observation? Maybe it would be possible to cancel somehow those requests made by the layer which is being removed, because they are already out of date?

Slawek
JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by jasonbeverage
Hi Jason,

I wrote a quick sample app to try to get the database pager to die consistently.

Source attached:
osgviewer_test.cpp

Along with my data for good measure:
test_dem.tif 
test_img.tif

For some reason, I get different results from my testing between debug and release, as follows:

debug:
1. Turn on wireframe (this is just to see paging more easily)
2. Hold down the right button on the mouse and move it forward and back, so that the model zooms in and out.
3. While zooming like this, hold down the '1' key. This is the key that re-creates the Map on keydown, so what we're doing is recreating the map many times.
4. Do this silly gesture with the mouse and the '1' key for some amount of time (10 seconds?)
5. Release everything and check to see if the database pager is dead.
6. Repeat if necessary.

release:
1. Turn on wireframe (this is just to see paging more easily)
2. Zoom in more closely to the terrain so stuff starts paging in.
3. Hold down the '1' key. No need to zoom in and out like in debug.
4. Eventually, you'll get a crash. I'm unsure of the status of the database pager in this case.

Sorry for the weird use cases, but unfortunately, this is the best thing i could come up with. It actually seems to be easier to reproduce in my application, but I can't attach that source, sorry. ;)

Thanks,
Jesse
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hey Jesse,

I just tested your app and can get the pager to "die".  The thread itself it not dying for me,it's stuck on a WaitForMultipleObjects call in OpenThreads::cooperativeWait in win32thread.cpp.

I would say this is not an osgEarth problem but a problem with the OSG DatabasePager itself.  I can also reproduce the problem using a regular VPB database (http://www.openscenegraph.org/data/earth_bayarea/earth.ive).

Could you try again in debug and when the pager "dies", attach to the process using Visual Studio, pause the app and verify that you are seeing the same thing?

Thanks,

Jason

On Wed, Jan 13, 2010 at 2:12 PM, JesseStimpson [via osgEarth] <[hidden email]> wrote:
Hi Jason,

I wrote a quick sample app to try to get the database pager to die consistently.

Source attached:
osgviewer_test.cpp

Along with my data for good measure:
test_dem.tif 
test_img.tif

For some reason, I get different results from my testing between debug and release, as follows:

debug:
1. Turn on wireframe (this is just to see paging more easily)
2. Hold down the right button on the mouse and move it forward and back, so that the model zooms in and out.
3. While zooming like this, hold down the '1' key. This is the key that re-creates the Map on keydown, so what we're doing is recreating the map many times.
4. Do this silly gesture with the mouse and the '1' key for some amount of time (10 seconds?)
5. Release everything and check to see if the database pager is dead.
6. Repeat if necessary.

release:
1. Turn on wireframe (this is just to see paging more easily)
2. Zoom in more closely to the terrain so stuff starts paging in.
3. Hold down the '1' key. No need to zoom in and out like in debug.
4. Eventually, you'll get a crash. I'm unsure of the status of the database pager in this case.

Sorry for the weird use cases, but unfortunately, this is the best thing i could come up with. It actually seems to be easier to reproduce in my application, but I can't attach that source, sorry. ;)

Thanks,
Jesse


View message @ http://n2.nabble.com/Long-time-of-layers-switching-tp3445055p4363618.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


gwaldron gwaldron
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Jason, this sounds suspiciously similar to the OpenThreads::Condition::broadcast() "bug" we encountered when developing the TaskService. OpenThreads::Block uses Condition, and DatabasePager uses Block. Condition::broadcast() hangs waiting for all the threads to wake up.

Also..someone did make a bug fix to OpenThreads Win32Condition not too long ago. Not sure if that fixes the broadcast() problem but it's worth investigating.

Glenn
Glenn Waldron / Pelican Mapping
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by slawowski
Hi Slawek,

I've been able to reproduce this issue and determined that the real bottleneck is the acquisition of the write lock on the map's mapDataMutex.  I've filed a ticket at http://www.osgearth.org/ticket/89 for this issue.  We'll have to do some rework to avoid this issue.

Thanks,

Jason

On Tue, Jan 12, 2010 at 2:14 PM, slawowski [via osgEarth] <[hidden email]> wrote:
Hi Jason,

Thanks for your clarifications. I have tested "sequential" mode once again and now I see a positive difference between this one and "standard" mode. Unfortunately, my program still hangs for more than just few seconds. Here are the details.

When I change the layer set, I remove all the existing layers and add new ones, even if they contain layers which were already present in the map (I know this could be optimized, but there may be cases in which all the layers are replaced). Initially I have only one raster layer from my local VPB database, so it loads quickly all necessary tiles even when I am close to earth. Then I add an online layer, for example OpenStreetMap. The program hangs for a second or two so it is acceptable. Then, I remove the online layer - I do it when I am close to earth and tiles from the online layer were still loading. This is the situation when the problem occurs - my program hangs for about 1-3 minutes, which is not a reasonable time.

Then I repeated this scenario, but looking at the earth from a much higher point, so that I could see the whole globe. In this case both adding and removing layers worked fast, the program did not hang for more than 1 second.

To sum up, it seems that in "sequential" mode especially time consuming is removing a layer which requested many tiles (because of view point set close to earth terrain). Do you agree with my observation? Maybe it would be possible to cancel somehow those requests made by the layer which is being removed, because they are already out of date?

Slawek



View message @ http://n2.nabble.com/Long-time-of-layers-switching-tp3445055p4293716.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by jasonbeverage
Jason,

jasonbeverage wrote
Could you try again in debug and when the pager "dies", attach to the process using Visual Studio, pause the app and verify that you are seeing the same thing?
Verified. I'm seeing the same thing.

If there's anything I can help with regarding this issue, please let me know. It's becoming a fairly large problem for us, and I'd like to contribute as much as possible.

Thanks,
Jesse
JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

In reply to this post by jasonbeverage
Hi Jason,

Hate to be a bother, but is there any news on this front? Are we deciding that it is in fact a bug in OSG and not osgEarth?

Also, how did you reproduce the problem with a VPB db? If I am able to reproduce with VPB, I'd be happy to post about it on the VPB board.

Thanks,
Jesse
jasonbeverage jasonbeverage
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi Jesse,

I was able to reproduce the issue using your sample app but instead of creating an osgEarth Map I was loading a VPB database from http://www.openscenegraph.org/data/earth_bayarea/earth.ive using osgDB::readNodeFile.

Try reproducing it using the VPB database and post it to the osg-users mailing list if you can.

Thanks,

Jason

On Thu, Jan 21, 2010 at 9:46 AM, JesseStimpson [via osgEarth] <[hidden email]> wrote:
Hi Jason,

Hate to be a bother, but is there any news on this front? Are we deciding that it is in fact a bug in OSG and not osgEarth?

Also, how did you reproduce the problem with a VPB db? If I am able to reproduce with VPB, I'd be happy to post about it on the VPB board.

Thanks,
Jesse


View message @ http://n2.nabble.com/Long-time-of-layers-switching-tp3445055p4433984.html
To start a new topic under osgEarth, email [hidden email]
To unsubscribe from osgEarth, click here.


JesseStimpson JesseStimpson
Reply | Threaded
Open this post in threaded view
|

Re: Long time of layers' switching

Hi Jason and Glenn,

After a lot of fighting with OSG's message board (it wasn't allowing me to post because it thought I had URLs in my message), I did end up making a post about the database pager dying... no one responded.

The point of this particular post is to 1) keep you updated and 2) express my gratitude for you and Glenn going above and beyond to help out me and the other osgEarth users. It really is amazing how fast you guys respond, and how graciously you put up with my ignorance. =)

Anyway, I'd like to offer to make a small donation to the Open Source project of your choice. (and osgEarth can be your choice if you do in fact take donations)

Thanks again,
Jesse
12