Paper available

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Paper available

Diva Canto
May be of interest to some on this list:
http://arxiv.org/abs/1508.04465

Lots of references to mantises and assorted OpenSim things.
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Quantifying and Replicating Open Simulator Performance (UNCLASSIFIED)

Maxwell, Douglas CIV USARMY RDECOM ARL (US)
Classification: UNCLASSIFIED
Caveats: NONE

We've had trouble replicating the successes advertised by other Grids and Open Simulator activities over the years.  The OSCC was an impressive achievement, however it took direct involvement and hand-tuning by Nebadon & others to help us get the MOSES grid in shape for last Spring's Federal Consortium for Virtual Worlds.  This isn't an indictment in any way of the people who worked on the high performance grids, simply a broad observation that grid deployment is complicated and the sprint-like development leaves documentation in an anemic shape.

The MOSES team has decided to take a deliberate, scientific approach to quantifying affects of some of these variables on performance.  The tangible result you can see are the recent three large patches where we provided fixes and updates in the stats collectors in the Open Simulator.  Using these new reliable statistics measures, we have begun publishing our findings in the following recent venues:

Summer Sim 2015 - these papers are behind a paywall, but it you contact me directly I can get you PDFs.
Conference Program: http://scs.org/documents/SummerSim15/SummerSim15%20Program%20Final.pdf

1)  Vertical Scalability Benchmarking in Three-Dimensional Virtual World Simulation
Abstract:  
"The United States military is investigating large-scale, realistic virtual world simulations to facilitate war-fighter training. As the simulation community strives towards meeting these military training objectives, methods must be developed and validated that measure scalability performance in these virtual world simulators. With such methods, the simulation community will be able to quantifiably compare scalability performance between system changes. This work contributes to the development and validation prerequisite by evaluating the effectiveness of commonly used system metrics to measure scalability in a three-dimensional virtual trainer. Specifically, the metrics of CPU utilization and simulation frames per second are evaluated for their effectiveness in vertical scalability benchmarking."

2)  Approach to Examine Efficacy of Game-Based and Virtual Simulation Training
Abstract:
"The United States Army has heavily leveraged, developed and expanded its use of virtual simulation training, as this class of simulation has been empirically demonstrated to be effective in the transfer of skills to the live environment. Game-based training, an alternative class of simulation, is characterized by its lower overhead and cost and potentially represents a less expensive alternative to virtual simulation training. In an effort to reduce the cost of training simulation, the U.S. Army has recently socialized the concept of potentially replacing select virtual simulation trainers with game-based simulations. While lowering the cost of simulation is a noble endeavor, the aforementioned concept requires further investigation as minimal empirical evidence exists regarding the effectiveness of game-based training, particularly at the collective echelon of training. In this paper, we lay the foundation to conduct an investigation of whether a game-based simulation may b
 e as equally an effective collective training apparatus as a higher-cost, higher-fidelity, virtual simulation. Specifically, we discuss a planned Training Effectiveness Evaluation (TEE) of both the Aviation Combined Arms Tactical Trainer (AVCATT) and a game-based aviation simulation that will empirically determine whether or not virtual training in the AVCATT could potentially be replaced by an equally effective, but less costly, game-based simulation. We discourse on our proposed design of experiment, which will utilize qualified Army aviators performing a tactical, collective mission in two discrete training treatments (AVCATT and a game based simulation) at Fort Rucker, Alabama."

Open Journal of Modeling and Simulation

3)  Analyzing Virtual World Region Fidelity on Scalability and Simulation Performance
http://www.scirp.org/journal/PaperDownload.aspx?paperID=58322

As you can see, we are not only making advancements with the Open Simulator, we are also testing our work with real soldiers.  It is important for you, the devs, to understand that the work you are doing is making a tangible difference.

v/r -doug

Dr. Douglas Maxwell
Science and Technology Manager
Virtual World Strategic Applications
U.S. Army Research Lab
Simulation & Training Technology Center (STTC)
(c) (407) 242-0209


-----Original Message-----
From: [hidden email] [mailto:[hidden email]] On Behalf Of Diva Canto
Sent: Wednesday, August 19, 2015 8:24 PM
To: [hidden email]
Subject: [Opensim-dev] Paper available

May be of interest to some on this list:
http://arxiv.org/abs/1508.04465

Lots of references to mantises and assorted OpenSim things.
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev

Classification: UNCLASSIFIED
Caveats: NONE


_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Quantifying and Replicating Open Simulator Performance (UNCLASSIFIED)

Diva Canto
Excellent, thanks for sharing these papers!Looks like they have much in
common with the one I sent out yesterday.

(as an aside, I recently started uploading preprints of my papers to
arXiv, in the period between they are accepted until they are actually
published. That way, a close-to-final version of the papers becomes
available to everyone without stepping on the publishers' copyrights...
this plus the automatic citation notifications of Google Schoolar makes
it a powerful advertising platform for academics!)

Anyway, performance and non-functional testing of distributed systems is
definitely "the lost world" of software engineering, and a lot needs to
happen in order to get it out of the dark age. OpenSim is a fantastic
framework for doing this kind of work!


On 8/20/2015 6:03 AM, Maxwell, Douglas CIV USARMY RDECOM (US) wrote:

> Classification: UNCLASSIFIED
> Caveats: NONE
>
> We've had trouble replicating the successes advertised by other Grids and Open Simulator activities over the years.  The OSCC was an impressive achievement, however it took direct involvement and hand-tuning by Nebadon & others to help us get the MOSES grid in shape for last Spring's Federal Consortium for Virtual Worlds.  This isn't an indictment in any way of the people who worked on the high performance grids, simply a broad observation that grid deployment is complicated and the sprint-like development leaves documentation in an anemic shape.
>
> The MOSES team has decided to take a deliberate, scientific approach to quantifying affects of some of these variables on performance.  The tangible result you can see are the recent three large patches where we provided fixes and updates in the stats collectors in the Open Simulator.  Using these new reliable statistics measures, we have begun publishing our findings in the following recent venues:
>
> Summer Sim 2015 - these papers are behind a paywall, but it you contact me directly I can get you PDFs.
> Conference Program: http://scs.org/documents/SummerSim15/SummerSim15%20Program%20Final.pdf
>
> 1)  Vertical Scalability Benchmarking in Three-Dimensional Virtual World Simulation
> Abstract:
> "The United States military is investigating large-scale, realistic virtual world simulations to facilitate war-fighter training. As the simulation community strives towards meeting these military training objectives, methods must be developed and validated that measure scalability performance in these virtual world simulators. With such methods, the simulation community will be able to quantifiably compare scalability performance between system changes. This work contributes to the development and validation prerequisite by evaluating the effectiveness of commonly used system metrics to measure scalability in a three-dimensional virtual trainer. Specifically, the metrics of CPU utilization and simulation frames per second are evaluated for their effectiveness in vertical scalability benchmarking."
>
> 2)  Approach to Examine Efficacy of Game-Based and Virtual Simulation Training
> Abstract:
> "The United States Army has heavily leveraged, developed and expanded its use of virtual simulation training, as this class of simulation has been empirically demonstrated to be effective in the transfer of skills to the live environment. Game-based training, an alternative class of simulation, is characterized by its lower overhead and cost and potentially represents a less expensive alternative to virtual simulation training. In an effort to reduce the cost of training simulation, the U.S. Army has recently socialized the concept of potentially replacing select virtual simulation trainers with game-based simulations. While lowering the cost of simulation is a noble endeavor, the aforementioned concept requires further investigation as minimal empirical evidence exists regarding the effectiveness of game-based training, particularly at the collective echelon of training. In this paper, we lay the foundation to conduct an investigation of whether a game-based simulation may
  b

>   e as equally an effective collective training apparatus as a higher-cost, higher-fidelity, virtual simulation. Specifically, we discuss a planned Training Effectiveness Evaluation (TEE) of both the Aviation Combined Arms Tactical Trainer (AVCATT) and a game-based aviation simulation that will empirically determine whether or not virtual training in the AVCATT could potentially be replaced by an equally effective, but less costly, game-based simulation. We discourse on our proposed design of experiment, which will utilize qualified Army aviators performing a tactical, collective mission in two discrete training treatments (AVCATT and a game based simulation) at Fort Rucker, Alabama."
>
> Open Journal of Modeling and Simulation
>
> 3)  Analyzing Virtual World Region Fidelity on Scalability and Simulation Performance
> http://www.scirp.org/journal/PaperDownload.aspx?paperID=58322
>
> As you can see, we are not only making advancements with the Open Simulator, we are also testing our work with real soldiers.  It is important for you, the devs, to understand that the work you are doing is making a tangible difference.
>
> v/r -doug
>
> Dr. Douglas Maxwell
> Science and Technology Manager
> Virtual World Strategic Applications
> U.S. Army Research Lab
> Simulation & Training Technology Center (STTC)
> (c) (407) 242-0209
>
>
> -----Original Message-----
> From: [hidden email] [mailto:[hidden email]] On Behalf Of Diva Canto
> Sent: Wednesday, August 19, 2015 8:24 PM
> To: [hidden email]
> Subject: [Opensim-dev] Paper available
>
> May be of interest to some on this list:
> http://arxiv.org/abs/1508.04465
>
> Lots of references to mantises and assorted OpenSim things.
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
> Classification: UNCLASSIFIED
> Caveats: NONE
>
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
>


_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Loading...