proposal: cleanup and break up region modules

classic Classic list List threaded Threaded
17 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

proposal: cleanup and break up region modules

Dr Scofield
i've been looking at where region modules live in our source tree --- OpenSim/Region/Modules and OpenSim/Region/Environment/Modules --- and how they get bundled:
  • modules in OpenSim/Region/Modules get their own private DLL
  • OpenSim/Region/Environment/Modules get lumped into ONE gigantic DLL
off the 3 modules living in OpenSim/Region/Modules 2 might be good candidate for forge project: python and SvnSerializer. the third really belongs to the Terrain region module and seems to contain the default terrain effects.

i think it would make sense to
  • have all region modules living in the same neighborhood (i'd prefer OpenSim/Region/Modules), the current layout is a bit confusing
  • break up the region module super-DLL so that each region module gets a DLL of its own
this would make the whole region module business much more, err, modular (and, who knows, perhaps we'd even have one day a build system like the "make menuconfig" one found in the linux kernel tree)

what is the general opinion on this? i'd be willing to undertake the refactoring (including tackling the prebuild.xml dragon).

    DrS/dirk
-- 
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/

_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

justincc
Dr Scofield wrote:

> i've been looking at where region modules live in our source tree ---
> OpenSim/Region/Modules and OpenSim/Region/Environment/Modules --- and
> how they get bundled:
>
>     * modules in OpenSim/Region/Modules get their own private DLL
>     * OpenSim/Region/Environment/Modules get lumped into ONE gigantic DLL
>
> off the 3 modules living in OpenSim/Region/Modules 2 might be good
> candidate for forge project: python and SvnSerializer. the third really
> belongs to the Terrain region module and seems to contain the default
> terrain effects.
>
> i think it would make sense to
>
>     * have all region modules living in the same neighborhood (i'd
>       prefer OpenSim/Region/Modules), the current layout is a bit confusing

Don't we need to make a distinction between 'service' modules (such as the REST module) and scene/environment modules
though?  The former are not attached to a scene (and may well never be concerned with scene code) while the latter are
very much scene related.

>     * break up the region module super-DLL so that each region module
>       gets a DLL of its own

At least on mono, I'm guessing that this would vastly expand build times.  Certainly at the moment, each invocation of
mono to build a new assembly appears to take far longer than building many files in a single dll.

I also tend to think of the current modules in OpenSim/Region/Environment/Modules as fairly core modules that one would
expect to be packaged together, and which it would be inconvenient to split up.

>
> this would make the whole region module business much more, err, modular
> (and, who knows, perhaps we'd even have one day a build system like the
> "make menuconfig" one found in the linux kernel tree)
>
> what is the general opinion on this? i'd be willing to undertake the
> refactoring (including tackling the prebuild.xml dragon).
>
>     DrS/dirk
>
> --
> dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
> SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev


--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Dr Scofield
Justin Clark-Casey wrote:

> Dr Scofield wrote:
>> i've been looking at where region modules live in our source tree ---
>> OpenSim/Region/Modules and OpenSim/Region/Environment/Modules --- and
>> how they get bundled:
>>
>>     * modules in OpenSim/Region/Modules get their own private DLL
>>     * OpenSim/Region/Environment/Modules get lumped into ONE gigantic DLL
>>
>> off the 3 modules living in OpenSim/Region/Modules 2 might be good
>> candidate for forge project: python and SvnSerializer. the third really
>> belongs to the Terrain region module and seems to contain the default
>> terrain effects.
>>
>> i think it would make sense to
>>
>>     * have all region modules living in the same neighborhood (i'd
>>       prefer OpenSim/Region/Modules), the current layout is a bit confusing
>
> Don't we need to make a distinction between 'service' modules (such as the REST module) and scene/environment modules
> though?  The former are not attached to a scene (and may well never be concerned with scene code) while the latter are
> very much scene related.

that would be another step. first i'd like to get the modules settled in the
same neighborhood.

>
>>     * break up the region module super-DLL so that each region module
>>       gets a DLL of its own
>
> At least on mono, I'm guessing that this would vastly expand build times.  Certainly at the moment, each invocation of
> mono to build a new assembly appears to take far longer than building many files in a single dll.

vastly?


>
> I also tend to think of the current modules in OpenSim/Region/Environment/Modules as fairly core modules that one would
> expect to be packaged together, and which it would be inconvenient to split up.

hmm...that kind of goes against the idea of them being modules, i'd think :-)


--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

justincc
Dr Scofield wrote:

> Justin Clark-Casey wrote:
>> Dr Scofield wrote:
>>> i've been looking at where region modules live in our source tree ---
>>> OpenSim/Region/Modules and OpenSim/Region/Environment/Modules --- and
>>> how they get bundled:
>>>
>>>     * modules in OpenSim/Region/Modules get their own private DLL
>>>     * OpenSim/Region/Environment/Modules get lumped into ONE gigantic DLL
>>>
>>> off the 3 modules living in OpenSim/Region/Modules 2 might be good
>>> candidate for forge project: python and SvnSerializer. the third really
>>> belongs to the Terrain region module and seems to contain the default
>>> terrain effects.
>>>
>>> i think it would make sense to
>>>
>>>     * have all region modules living in the same neighborhood (i'd
>>>       prefer OpenSim/Region/Modules), the current layout is a bit confusing
>> Don't we need to make a distinction between 'service' modules (such as the REST module) and scene/environment modules
>> though?  The former are not attached to a scene (and may well never be concerned with scene code) while the latter are
>> very much scene related.
>
> that would be another step. first i'd like to get the modules settled in the
> same neighborhood.
>
>>>     * break up the region module super-DLL so that each region module
>>>       gets a DLL of its own
>> At least on mono, I'm guessing that this would vastly expand build times.  Certainly at the moment, each invocation of
>> mono to build a new assembly appears to take far longer than building many files in a single dll.
>
> vastly?

Maybe this is a slight exaggeration but I'm anticipating that the build time would be much longer.  I'm happy to be
proved wrong on this point.  Maybe this seems a minor thing but long build times are such a pain in the ass.

>
>
>> I also tend to think of the current modules in OpenSim/Region/Environment/Modules as fairly core modules that one would
>> expect to be packaged together, and which it would be inconvenient to split up.
>
> hmm...that kind of goes against the idea of them being modules, i'd think :-)
>
>

Does it?  Isn't this just a convenient way of packaging the core modules that almost everybody is going to need to run
their OpenSim (though I would admit that some of the modules in there arguably aren't core...)

I feel that what would be really useful is a core mechanism for enabling/disabling modules which doesn't rely on the
module itself having that option.  This would be simpler and might make the question of whether there are lots of
modules bundled in a single dll moot (since you can just enable/disable them with separate config entries - and storage
space for dlls is cheap :-)

Which in itself leads me on to the question of splitting up the massive monolithic config file.  I don't know about
other people, but I find it a pain to deal with.  I feel that these kinds of changes would be really valuable whilst
rearranging the module spaces is kind of nice to have but doesn't seem the most pressing issue right now.  But that's
just my opinion :-)

--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Sean Dague-2
Justin Clark-Casey wrote:

> Dr Scofield wrote:
>> Justin Clark-Casey wrote:
>>> Dr Scofield wrote:
>>>> i've been looking at where region modules live in our source tree ---
>>>> OpenSim/Region/Modules and OpenSim/Region/Environment/Modules --- and
>>>> how they get bundled:
>>>>
>>>>     * modules in OpenSim/Region/Modules get their own private DLL
>>>>     * OpenSim/Region/Environment/Modules get lumped into ONE gigantic DLL
>>>>
>>>> off the 3 modules living in OpenSim/Region/Modules 2 might be good
>>>> candidate for forge project: python and SvnSerializer. the third really
>>>> belongs to the Terrain region module and seems to contain the default
>>>> terrain effects.
>>>>
>>>> i think it would make sense to
>>>>
>>>>     * have all region modules living in the same neighborhood (i'd
>>>>       prefer OpenSim/Region/Modules), the current layout is a bit confusing
>>> Don't we need to make a distinction between 'service' modules (such as the REST module) and scene/environment modules
>>> though?  The former are not attached to a scene (and may well never be concerned with scene code) while the latter are
>>> very much scene related.
>> that would be another step. first i'd like to get the modules settled in the
>> same neighborhood.
>>
>>>>     * break up the region module super-DLL so that each region module
>>>>       gets a DLL of its own
>>> At least on mono, I'm guessing that this would vastly expand build times.  Certainly at the moment, each invocation of
>>> mono to build a new assembly appears to take far longer than building many files in a single dll.
>> vastly?
>
> Maybe this is a slight exaggeration but I'm anticipating that the build time would be much longer.  I'm happy to be
> proved wrong on this point.  Maybe this seems a minor thing but long build times are such a pain in the ass.
It may add a bit of time, but I don't think it would be much.  If going
after build times is a goal, I think that's probably a seperate topic.
Nant (like ant) does a lot of dumb things that waste time.

>>> I also tend to think of the current modules in OpenSim/Region/Environment/Modules as fairly core modules that one would
>>> expect to be packaged together, and which it would be inconvenient to split up.
>> hmm...that kind of goes against the idea of them being modules, i'd think :-)
>>
>>
>
> Does it?  Isn't this just a convenient way of packaging the core modules that almost everybody is going to need to run
> their OpenSim (though I would admit that some of the modules in there arguably aren't core...)
>
> I feel that what would be really useful is a core mechanism for enabling/disabling modules which doesn't rely on the
> module itself having that option.  This would be simpler and might make the question of whether there are lots of
> modules bundled in a single dll moot (since you can just enable/disable them with separate config entries - and storage
> space for dlls is cheap :-)
I'd be much more of a fan of having each module a seperate dll.  Files
are cheap too. :)  And that makes it very clear to people what they are
loading, and what they aren't loading.

> Which in itself leads me on to the question of splitting up the massive monolithic config file.  I don't know about
> other people, but I find it a pain to deal with.  I feel that these kinds of changes would be really valuable whilst
> rearranging the module spaces is kind of nice to have but doesn't seem the most pressing issue right now.  But that's
> just my opinion :-)

I'd be a fan of that as well.  More importantly, I think we need to get
down to 1 config system and 1 plugin system for the project.  Right now
I think we're still at 2 or 3 per, which makes it painful all around.

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Tleiades

> I'd be much more of a fan of having each module a seperate dll.  Files
> are cheap too. :)  And that makes it very clear to people what they are
> loading, and what they aren't loading.
>

(On the fear of talking about performance prematurely)
Won't that cause problems for the JIT'er?

Normally access to member variables gets optimized away into a direct
memory access rather than invocation of a JSR. If I recall correctly
this optimization does not work for dynamically loaded assemblies.



_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Dr Scofield
In reply to this post by Sean Dague-2
Tleiades wrote:

>> I'd be much more of a fan of having each module a seperate dll.  Files
>> are cheap too. :)  And that makes it very clear to people what they are
>> loading, and what they aren't loading.
>>
>
> (On the fear of talking about performance prematurely)
> Won't that cause problems for the JIT'er?
>
> Normally access to member variables gets optimized away into a direct
> memory access rather than invocation of a JSR. If I recall correctly
> this optimization does not work for dynamically loaded assemblies.

well, if that's the case, then it's not working currently either :-) currently
we lump all region modules into one large super DLL and load that dynamically.

--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Tleiades


> Date: Thu, 29 Jan 2009 08:31:48 +0100
> From: [hidden email]
> To: [hidden email]
> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
>
> Tleiades wrote:
> >> I'd be much more of a fan of having each module a seperate dll. Files
> >> are cheap too. :) And that makes it very clear to people what they are
> >> loading, and what they aren't loading.
> >>
> >
> > (On the fear of talking about performance prematurely)
> > Won't that cause problems for the JIT'er?
> >
> > Normally access to member variables gets optimized away into a direct
> > memory access rather than invocation of a JSR. If I recall correctly
> > this optimization does not work for dynamically loaded assemblies.
>
> well, if that's the case, then it's not working currently either :-) currently
> we lump all region modules into one large super DLL and load that dynamically.

I  guess what I'm saying is that dll files are not as cheap as it is being implied. Having an application dynamicallly load a large number of dll's at runtime, is less efficient that loading a few large dll's during load time. The JIT'ed code will be less efficient and loadtime of the application will increase. While load time may not be a big issue, I believe it is best to give the JIT engine the best working condition.

As I understand it the JIT engine and assembly loader have been designed based on a use pattern which states: "Most assemblies will be loaded during application load time, and only few assemblies will be loaded at a latter stage", I definately know this to be a fact for the MS .Net engine, but I don't know if that is also the case for Mono, although I believe it will be safe to assume so.

>
> --
> dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
> SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev


check out the rest of the Windows Live™. More than mail–Windows Live™ goes way beyond your inbox. More than messages
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Dr Scofield
Tleiades Lauridsen wrote:

>
>
>> Date: Thu, 29 Jan 2009 08:31:48 +0100
>> From: [hidden email]
>> To: [hidden email]
>> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
>>
>> Tleiades wrote:
>> >> I'd be much more of a fan of having each module a seperate dll. Files
>> >> are cheap too. :) And that makes it very clear to people what they are
>> >> loading, and what they aren't loading.
>> >>
>> >
>> > (On the fear of talking about performance prematurely)
>> > Won't that cause problems for the JIT'er?
>> >
>> > Normally access to member variables gets optimized away into a direct
>> > memory access rather than invocation of a JSR. If I recall correctly
>> > this optimization does not work for dynamically loaded assemblies.
>>
>> well, if that's the case, then it's not working currently either :-)
> currently
>> we lump all region modules into one large super DLL and load that
> dynamically.
>
> I  guess what I'm saying is that dll files are not as cheap as it is
> being implied. Having an application dynamicallly load a large number of
> dll's at runtime, is less efficient that loading a few large dll's
> during load time. The JIT'ed code will be less efficient and loadtime of
> the application will increase. While load time may not be a big issue, I
> believe it is best to give the JIT engine the best working condition.

we are always loading at runtime --- or let me ask the other way round: what do
you define as "loadtime"?


>
> As I understand it the JIT engine and assembly loader have been designed
> based on a use pattern which states: "Most assemblies will be loaded
> during application load time, and only few assemblies will be loaded at
> a latter stage",

well, we are loading at runtime then.

--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Stefan Andersson
I think that some confusion stems from the fact that it's a difference between referencing the assembly, and dynamically instancing classes out of it.
 
If you reference classes directly, as in
 
mything = new Thing();
 
where thing could be a module, albeit _referenced_, not _loaded_
 
is a huge difference to
 
mything = loadedAssembly.Instantiate( "Thing" );

in the former, the jit compiler can do all kinds of neat optimizations when referencing classes in the Assembly, since it has both the referenced component and the referrer at hand. It can also inline and optimize away type validity checks and trusted domain checks.
 
In the latter, it really can't, and need to wrap everything in prudent encapsulation, protection and indirection.

Optimization points aside, I would hate for us to expand the core modules into separate projects just because we can; I think we would have to add 20 more projects to the solution and that would just suck - especially now that we have cleaned out so many. Some of the these new projects will have just a handful of classes, which is just an absurd waste of resources for something that should be used in 95% of the setups.
 
I actually don't know how mono addins work, but in the .net framework individual classes are referenced by assembly and class if loaded at runtime, so for examble
 
authModule= "MyAuthLib.dll, MyAuthenticator" would reference the class MyAuthenticator in the dll MyAuthLib.dll - which means that you can lump as many of those together in an dll that you want to.
 
so we should make sure we can do something liket that, and instead try to lump modules that share a common domain together.
 
The only two reasons to ever split an assembly into two is from
* references being too different (if you want to re-use them separately)
* encapsulation and security issues (actuallly USING internal as it was intended)
 
So, what I'm saying is that we should strive towards a situation where the core modules are bundled into a minimal set of assemblies, and the rest put on forge.

Best regards,
Stefan Andersson
Tribal Media AB

> Date: Thu, 29 Jan 2009 11:28:19 +0100

> From: [hidden email]
> To: [hidden email]
> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
>
> Tleiades Lauridsen wrote:
> >
> >
> >> Date: Thu, 29 Jan 2009 08:31:48 +0100
> >> From: [hidden email]
> >> To: [hidden email]
> >> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
> >>
> >> Tleiades wrote:
> >> >> I'd be much more of a fan of having each module a seperate dll. Files
> >> >> are cheap too. :) And that makes it very clear to people what they are
> >> >> loading, and what they aren't loading.
> >> >>
> >> >
> >> > (On the fear of talking about performance prematurely)
> >> > Won't that cause problems for the JIT'er?
> >> >
> >> > Normally access to member variables gets optimized away into a direct
> >> > memory access rather than invocation of a JSR. If I recall correctly
> >> > this optimization does not work for dynamically loaded assemblies.
> >>
> >> well, if that's the case, then it's not working currently either :-)
> > currently
> >> we lump all region modules into one large super DLL and load that
> > dynamically.
> >
> > I guess what I'm saying is that dll files are not as cheap as it is
> > being implied. Having an application dynamicallly load a large number of
> > dll's at runtime, is less efficient that loading a few large dll's
> > during load time. The JIT'ed code will be less efficient and loadtime of
> > the application will increase. While load time may not be a big issue, I
> > believe it is best to give the JIT engine the best working condition.
>
> we are always loading at runtime --- or let me ask the other way round: what do
> you define as "loadtime"?
>
>
> >
> > As I understand it the JIT engine and assembly loader have been designed
> > based on a use pattern which states: "Most assemblies will be loaded
> > during application load time, and only few assemblies will be loaded at
> > a latter stage",
>
> well, we are loading at runtime then.
>
> --
> dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
> SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev


_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

mw-2
Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.

While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.

And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.

Stefan Andersson <[hidden email]> wrote:
I think that some confusion stems from the fact that it's a difference between referencing the assembly, and dynamically instancing classes out of it.
 
If you reference classes directly, as in
 
mything = new Thing();
 
where thing could be a module, albeit _referenced_, not _loaded_
 
is a huge difference to
 
mything = loadedAssembly.Instantiate( "Thing" );

in the former, the jit compiler can do all kinds of neat optimizations when referencing classes in the Assembly, since it has both the referenced component and the referrer at hand. It can also inline and optimize away type validity checks and trusted domain checks.
 
In the latter, it really can't, and need to wrap everything in prudent encapsulation, protection and indirection.

Optimization points aside, I would hate for us to expand the core modules into separate projects just because we can; I think we would have to add 20 more projects to the solution and that would just suck - especially now that we have cleaned out so many. Some of the these new projects will have just a handful of classes, which is just an absurd waste of resources for something that should be used in 95% of the setups.
 
I actually don't know how mono addins work, but in the .net framework individual classes are referenced by assembly and class if loaded at runtime, so for examble
 
authModule= "MyAuthLib.dll, MyAuthenticator" would reference the class MyAuthenticator in the dll MyAuthLib.dll - which means that you can lump as many of those together in an dll that you want to.
 
so we should make sure we can do something liket that, and instead try to lump modules that share a common domain together.
 
The only two reasons to ever split an assembly into two is from
* references being too different (if you want to re-use them separately)
* encapsulation and security issues (actuallly USING internal as it was intended)
 
So, what I'm saying is that we should strive towards a situation where the core modules are bundled into a minimal set of assemblies, and the rest put on forge.

Best regards,
Stefan Andersson
Tribal Media AB

> Date: Thu, 29 Jan 2009 11:28:19 +0100

> From: [hidden email]
> To: [hidden email]
> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
>
> Tleiades Lauridsen wrote:
> >
> >
> >> Date: Thu, 29 Jan 2009 08:31:48 +0100
> >> From: [hidden email]
> >> To: [hidden email]
> >> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
> >>
> >> Tleiades wrote:
> >> >> I'd be much more of a fan of having each module a seperate dll. Files
> >> >> are cheap too. :) And that makes it very clear to people what they are
> >> >> loading, and what they aren't loading.
> >> >>
> >> >
> >> > (On the fear of talking about performance prematurely)
> >> > Won't that cause problems for the JIT'er?
> >> >
> >> > Normally access to member variables gets optimized away into a direct
> >> > memory access rather than invocation of a JSR. If I recall correctly
> >> > this optimization does not work for dynamically loaded assemblies.
> >>
> >> well, if that's the case, then it's not working currently either :-)
> > currently
> >> we lump all region modules into one large super DLL and load that
> > dynamically.
> >
> > I guess what I'm saying is that dll files are not as cheap as it is
> > being implied. Having an application dynamicallly load a large number of
> > dll's at runtime, is less efficient that loading a few large dll's
> > during load time. The JIT'ed code will be less efficient and loadtime of
> > the application will increase. While load time may not be a big issue, I
> > believe it is best to give the JIT engine the best working condition.
>
> we are always loading at runtime --- or let me ask the other way round: what do
> you define as "loadtime"?
>
>
> >
> > As I understand it the JIT engine and assembly loader have been designed
> > based on a use pattern which states: "Most assemblies will be loaded
> > during application load time, and only few assemblies will be loaded at
> > a latter stage",
>
> well, we are loading at runtime then.
>
> --
> dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
> SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev

_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev


_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Sean Dague-2
MW wrote:
> Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.
>
> While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.
>
> And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.

Ah, but the problem is then those all need to release at the same time.
 One of the reasons to get us into seperate dlls per module is so that
the IRC module could be updated and distributed without pushing out the
rest of the dll.

It would mean that we would easily be able to replace implementations
for various core modules without needing to touch the rest.  For
instance, when diva started to tackle the world map issue, she not only
had to build her own world map, but to change the one in the core to not
load.  Had we been under this model, she wouldn't have needed to do that.

Don't fear the projects, more projects don't hurt you. :)

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Cristina Videira Lopes
In reply to this post by mw-2
My experience with RESTComms, and having looked at the other modules in there, is that most of those aren't really "modules" as in "optional components", but as "the reference implementation of a required interface that can be replaced with another implementation". For that reason they can be seen as belonging to core, and be bundled in one single dll.

Although I would prefer to see each one of them in a separate dll, I think MW's proposal is perfectly reasonable. It's a good idea to separate these modules from, at least, OpenSim.Region.Environment.dll, if not from each other.

MW wrote:
Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.

While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.

And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.

Stefan Andersson [hidden email] wrote:
I think that some confusion stems from the fact that it's a difference between referencing the assembly, and dynamically instancing classes out of it.
 
If you reference classes directly, as in
 
mything = new Thing();
 
where thing could be a module, albeit _referenced_, not _loaded_
 
is a huge difference to
 
mything = loadedAssembly.Instantiate( "Thing" );

in the former, the jit compiler can do all kinds of neat optimizations when referencing classes in the Assembly, since it has both the referenced component and the referrer at hand. It can also inline and optimize away type validity checks and trusted domain checks.
 
In the latter, it really can't, and need to wrap everything in prudent encapsulation, protection and indirection.

Optimization points aside, I would hate for us to expand the core modules into separate projects just because we can; I think we would have to add 20 more projects to the solution and that would just suck - especially now that we have cleaned out so many. Some of the these new projects will have just a handful of classes, which is just an absurd waste of resources for something that should be used in 95% of the setups.
 
I actually don't know how mono addins work, but in the .net framework individual classes are referenced by assembly and class if loaded at runtime, so for examble
 
authModule= "MyAuthLib.dll, MyAuthenticator" would reference the class MyAuthenticator in the dll MyAuthLib.dll - which means that you can lump as many of those together in an dll that you want to.
 
so we should make sure we can do something liket that, and instead try to lump modules that share a common domain together.
 
The only two reasons to ever split an assembly into two is from
* references being too different (if you want to re-use them separately)
* encapsulation and security issues (actuallly USING internal as it was intended)
 
So, what I'm saying is that we should strive towards a situation where the core modules are bundled into a minimal set of assemblies, and the rest put on forge.

Best regards,
Stefan Andersson
Tribal Media AB

> Date: Thu, 29 Jan 2009 11:28:19 +0100
> From: [hidden email]
> To: [hidden email]
> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
>
> Tleiades Lauridsen wrote:
> >
> >
> >> Date: Thu, 29 Jan 2009 08:31:48 +0100
> >> From: [hidden email]
> >> To: [hidden email]
> >> Subject: Re: [Opensim-dev] proposal: cleanup and break up region modules
> >>
> >> Tleiades wrote:
> >> >> I'd be much more of a fan of having each module a seperate dll. Files
> >> >> are cheap too. :) And that makes it very clear to people what they are
> >> >> loading, and what they aren't loading.
> >> >>
> >> >
> >> > (On the fear of talking about performance prematurely)
> >> > Won't that cause problems for the JIT'er?
> >> >
> >> > Normally access to member variables gets optimized away into a direct
> >> > memory access rather than invocation of a JSR. If I recall correctly
> >> > this optimization does not work for dynamically loaded assemblies.
> >>
> >> well, if that's the case, then it's not working currently either :-)
> > currently
> >> we lump all region modules into one large super DLL and load that
> > dynamically.
> >
> > I guess what I'm saying is that dll files are not as cheap as it is
> > being implied. Having an application dynamicallly load a large number of
> > dll's at runtime, is less efficient that loading a few large dll's
> > during load time. The JIT'ed code will be less efficient and loadtime of
> > the application will increase. While load time may not be a big issue, I
> > believe it is best to give the JIT engine the best working condition.
>
> we are always loading at runtime --- or let me ask the other way round: what do
> you define as "loadtime"?
>
>
> >
> > As I understand it the JIT engine and assembly loader have been designed
> > based on a use pattern which states: "Most assemblies will be loaded
> > during application load time, and only few assemblies will be loaded at
> > a latter stage",
>
> well, we are loading at runtime then.
>
> --
> dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
> SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
> RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> https://lists.berlios.de/mailman/listinfo/opensim-dev

_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev


_______________________________________________ Opensim-dev mailing list [hidden email] https://lists.berlios.de/mailman/listinfo/opensim-dev


_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

justincc
In reply to this post by Sean Dague-2
Sean Dague wrote:

> MW wrote:
>> Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.
>>
>> While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.
>>
>> And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.
>
> Ah, but the problem is then those all need to release at the same time.
>  One of the reasons to get us into seperate dlls per module is so that
> the IRC module could be updated and distributed without pushing out the
> rest of the dll.

I would actually argue that in this particular case the IRC module is not a core module and should be in the forge.

>
> It would mean that we would easily be able to replace implementations
> for various core modules without needing to touch the rest.  For
> instance, when diva started to tackle the world map issue, she not only
> had to build her own world map, but to change the one in the core to not
> load.  Had we been under this model, she wouldn't have needed to do that.

This would still mean that people would have to move the dll being included in core so that it was no longer loaded by
the system.  To me, this seems no less of a task than having a configurable list of modules that are loaded.  Having to
move dlls also means that it isn't easy to pass configuration files around - instead you have to give instructions or
scripts for moving around modules in the filesystem.

>
> Don't fear the projects, more projects don't hurt you. :)

But in this case they do, in terms of cluttering up the solution with lots and lots of projects that are all used anyway
in almost all OpenSim installations and possible in terms of increasing build times (I admit that I may be paranoid on
this point and there might be ways to fix this via nant changes).  It will also increase the effort required for
refactoring existing code into modules, since creating a new project takes more time.

--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Sean Dague-2
Justin Clark-Casey wrote:

> Sean Dague wrote:
>> MW wrote:
>>> Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.
>>>
>>> While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.
>>>
>>> And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.
>> Ah, but the problem is then those all need to release at the same time.
>>  One of the reasons to get us into seperate dlls per module is so that
>> the IRC module could be updated and distributed without pushing out the
>> rest of the dll.
>
> I would actually argue that in this particular case the IRC module is not a core module and should be in the forge.
But that's really saying 2 things.  It should be a seperate dll, and it
should be in a seperate source tree.  I'm personally not a fan of doing
both of those things at once, because they are seperate things, and it's
easy to forget that.

>> It would mean that we would easily be able to replace implementations
>> for various core modules without needing to touch the rest.  For
>> instance, when diva started to tackle the world map issue, she not only
>> had to build her own world map, but to change the one in the core to not
>> load.  Had we been under this model, she wouldn't have needed to do that.
>
> This would still mean that people would have to move the dll being included in core so that it was no longer loaded by
> the system.  To me, this seems no less of a task than having a configurable list of modules that are loaded.  Having to
> move dlls also means that it isn't easy to pass configuration files around - instead you have to give instructions or
> scripts for moving around modules in the filesystem.
This comes back to the addins question.  Having seen this work pretty
well with gnome-do, I think that gets addressed seperately, including
the ability to notify you of new upstream modules and download them.

>> Don't fear the projects, more projects don't hurt you. :)
>
> But in this case they do, in terms of cluttering up the solution with lots and lots of projects that are all used anyway
> in almost all OpenSim installations and possible in terms of increasing build times (I admit that I may be paranoid on
> this point and there might be ways to fix this via nant changes).  It will also increase the effort required for
> refactoring existing code into modules, since creating a new project takes more time.

So we should put all the database drivers into a single dll?  As a Linux
user I'm pained far more by filesystem cluster and mismatch of classname
to .cs file, or dll to directory.

We've already decided that seperation and organization is needed, look
at all the structure in the OpenSim/Region/Environment/Modules directory
tree.

That being said, this isn't an all or nothing approach.  I'd say step
one, lets get the modules into a seperate dll (like diva suggested).
Then we can experiment with some level break up and see if it does cause
the end of the world. :)  If so, we can quickly go back.

Honestly, it should reduce build times.  As a change in 1 module isn't
going to require rebuild on everything.

        -Sean

--
Sean Dague / Neas Bade
[hidden email]
http://dague.net



_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev

signature.asc (260 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

Dr Scofield
Sean Dague wrote:
[...]
>
> That being said, this isn't an all or nothing approach.  I'd say step
> one, lets get the modules into a seperate dll (like diva suggested).
> Then we can experiment with some level break up and see if it does cause
> the end of the world. :)  If so, we can quickly go back.

that sounds like a good approach: move everything into
OpenSim.Region.CoreModules (as MW suggested) and then take it (or not) from
there with regards to separate DLLs.



--
dr dirk husemann ---- virtual worlds research ---- ibm zurich research lab
SL: dr scofield ---- [hidden email] ---- http://xyzzyxyzzy.net/
RL: [hidden email] - +41 44 724 8573 - http://www.zurich.ibm.com/~hud/
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: proposal: cleanup and break up region modules

justincc
In reply to this post by Sean Dague-2
Sean Dague wrote:

> Justin Clark-Casey wrote:
>> Sean Dague wrote:
>>> MW wrote:
>>>> Putting aside the optimisations for now, as I think that is a different question. As if we are going to have a dynamic module system then those issues come with it.
>>>>
>>>> While I think having every module in a separate dll/project is too much. As Stefan said I think we have just about enough projects in the core as it is. But I can see a case for moving all the modules that are currently in OpenSim.Region.Environment, into a single separate project/dll called something like OpenSim.Region.CoreModules.
>>>>
>>>> And if we could as Justin said, impove the module loading system, so that the operator can turn on and off the loading/usage of any modules in a single dll.
>>> Ah, but the problem is then those all need to release at the same time.
>>>  One of the reasons to get us into seperate dlls per module is so that
>>> the IRC module could be updated and distributed without pushing out the
>>> rest of the dll.
>> I would actually argue that in this particular case the IRC module is not a core module and should be in the forge.
>
> But that's really saying 2 things.  It should be a seperate dll, and it
> should be in a seperate source tree.  I'm personally not a fan of doing
> both of those things at once, because they are seperate things, and it's
> easy to forget that.

This is the case if we have a separate assembly for all modules, but if one has a single dll for all core modules then
they really aren't separate things.  If one finds oneself wanting to update a single module that doesn't provide
functionality that almost everybody will use (such as irc), then that module is not core and should be in its own
assembly.  And non-core modules in their own assemblies shouldn't really be shipped with core OpenSim in the long term -
they should be in the forge or hosted elsewhere.

>
>>> It would mean that we would easily be able to replace implementations
>>> for various core modules without needing to touch the rest.  For
>>> instance, when diva started to tackle the world map issue, she not only
>>> had to build her own world map, but to change the one in the core to not
>>> load.  Had we been under this model, she wouldn't have needed to do that.
>> This would still mean that people would have to move the dll being included in core so that it was no longer loaded by
>> the system.  To me, this seems no less of a task than having a configurable list of modules that are loaded.  Having to
>> move dlls also means that it isn't easy to pass configuration files around - instead you have to give instructions or
>> scripts for moving around modules in the filesystem.
>
> This comes back to the addins question.  Having seen this work pretty
> well with gnome-do, I think that gets addressed seperately, including
> the ability to notify you of new upstream modules and download them.

Yes, there may well be some interaction here with how mono.addins wants to work.

>
>>> Don't fear the projects, more projects don't hurt you. :)
>> But in this case they do, in terms of cluttering up the solution with lots and lots of projects that are all used anyway
>> in almost all OpenSim installations and possible in terms of increasing build times (I admit that I may be paranoid on
>> this point and there might be ways to fix this via nant changes).  It will also increase the effort required for
>> refactoring existing code into modules, since creating a new project takes more time.
>
> So we should put all the database drivers into a single dll?  As a Linux
> user I'm pained far more by filesystem cluster and mismatch of classname
> to .cs file, or dll to directory.

I'm not advocating it, but why not if they're all core supported databases?  One reason is that they all contain modules
which implement the same set of interfaces, so one has to choose between them.  At the moment one of the only way we
have to do this is by specifying dlls in OpenSim.ini (the other way is having the modules enable/disable depending on
passed in config, which is nasty).  If there were another way to specify enable/disable then one could put them all in a
single dll.

One can go the other way and say that why shouldn't we modularize more and put each individual database service (asset,
inventory, etc.) into its own module, so that we end up with 21 of them (not counting nhibernate stuff)?  After all,
people might want to host different services on different 'databases' (e.g. assets as files in a filesystem, user data
in LDAP and inventory data in MySQL).  You still end up with the issue of how to selectively enable/disable stuff
whether you put them in separate dlls or all in one.

>
> We've already decided that seperation and organization is needed, look
> at all the structure in the OpenSim/Region/Environment/Modules directory
> tree.
>
> That being said, this isn't an all or nothing approach.  I'd say step
> one, lets get the modules into a seperate dll (like diva suggested).
> Then we can experiment with some level break up and see if it does cause
> the end of the world. :)  If so, we can quickly go back.

I agree with what other people have said - these would be better off as a separate dll.  And by all means experiment
once this is done, but I would really not want to see us slide into something that more people have expressed concern
about than support for, at least not without seeing some more convincing arguments/evidence that the benefits outweigh
the cons (or that the cons are unfounded).

--
justincc
Justin Clark-Casey
http://justincc.wordpress.com
_______________________________________________
Opensim-dev mailing list
[hidden email]
https://lists.berlios.de/mailman/listinfo/opensim-dev
Loading...