Introduction of a sitemap style system for users and land

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Introduction of a sitemap style system for users and land

Alan Johnston
I did not receive the reply I think I set my options wrong in the mailman but I did see the reply in the online archives, Haravikk  The point of sitemaps is you submit then to a search engine for a search engine to have a set of URLs for them to look up and get the information from. Its a map to the site so to speak, this is good for such things as social sites, that wont have links on the main site that point to every possible page there is, as an example if you go to https://www.allvirtualsocial.com you will not see my post I posted on there a few weeks ago or the post about me creating android apps that are now available on the google play, but if you look at the sitemap https://www.allvirtualsocial.com/sitemap.xml you will see the link in there, although search engines dont use them for the main system to provide ranking, they use it to know where to crawl on the site, especially for pages that are not that obvious to “spiders”, so site maps are more common than people think, but are used primarily for search engines, this is why I have thought about this a lot. The concept would be to provide the search engine with the information it needs to provide search results to a set type, and would make be able to provide the nest answers it can give. I have a lot of work to do on the system including the bots, I’m thinking of making a 3 bot system 1 that does sites in the general sense, I.e. my social site, grids websites, news sites like Hyper Grid News, Blog sites related to virtual worlds and so forth, one bot to do site map collection for such sites, and another bot to do the land and users maps in grids, I’m going to install OS on my system some point today and check out some things on the system, and going to make a PHP sitemap Implementation of landmaps. And look at something when it comes to usermaps, though I would need to speak to someone within OS and Viewers to see if a thought is possible though I’m doubtful, this is something I am hopeing that can either be a module (and someone can help me create a opensource module of this) or even have it as a core integration, and have it with some config options so then people can choose to opt in or out of it.
The search engine is a long way off from being finished, I have yet to fully create the bots, I’m working on the first bot, and working on making it so it follows robots.txt and robots META tags, and working on the back end the options of what to use for the back end, elastic search, DB system like MySQL or alike, or even a custom engine design, I have a lot of work ahead of me and I’m working hard on it,  I’m also working hard on the page ranking algorithm aswell which will be hard to do. If people are wanting to help with this I would be very happy to accept help on this project in any form.

Sent from Mail for Windows 10

_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: Introduction of a sitemap style system for users and land

Cinder Roxley
Have you seen the functionality in Core that drives MetaverseInk’s web
search already?
http://www.metaverseink.com/blog/news/metaverse-ink-search-for-opensims/ I
would suggest leveraging that for land instead. Crawling seems like a waste
of resources when a grid could simply push the data to you in a format
easily consumable. A module could be developed that pushes other updates to
the search as well. (Does events work in Core? I don’t remember.) Failing
that, a module that provides the information via a common api makes a lot
more sense than scraping webpages.

I wrote a search module a while back, I could open source it and you could
build upon it with an external api that would provide consumers with
relevant data in json or whatnot. Although, I still consider a push data
model to the the more efficient design than poll and scrape.

On July 4, 2018 at 7:56:44 AM, Alan Johnston (
[hidden email]) wrote:

I did not receive the reply I think I set my options wrong in the mailman
but I did see the reply in the online archives, Haravikk The point of
sitemaps is you submit then to a search engine for a search engine to have
a set of URLs for them to look up and get the information from. Its a map
to the site so to speak, this is good for such things as social sites, that
wont have links on the main site that point to every possible page there
is, as an example if you go to https://www.allvirtualsocial.com you will
not see my post I posted on there a few weeks ago or the post about me
creating android apps that are now available on the google play, but if you
look at the sitemap https://www.allvirtualsocial.com/sitemap.xml you will
see the link in there, although search engines dont use them for the main
system to provide ranking, they use it to know where to crawl on the site,
especially for pages that are not that obvious to “spiders”, so site maps
are more common than people think, but are used primarily for search
engines, this is why I have thought about this a lot. The concept would be
to provide the search engine with the information it needs to provide
search results to a set type, and would make be able to provide the nest
answers it can give. I have a lot of work to do on the system including the
bots, I’m thinking of making a 3 bot system 1 that does sites in the
general sense, I.e. my social site, grids websites, news sites like Hyper
Grid News, Blog sites related to virtual worlds and so forth, one bot to do
site map collection for such sites, and another bot to do the land and
users maps in grids, I’m going to install OS on my system some point today
and check out some things on the system, and going to make a PHP sitemap
Implementation of landmaps. And look at something when it comes to
usermaps, though I would need to speak to someone within OS and Viewers to
see if a thought is possible though I’m doubtful, this is something I am
hopeing that can either be a module (and someone can help me create a
opensource module of this) or even have it as a core integration, and have
it with some config options so then people can choose to opt in or out of
it.
The search engine is a long way off from being finished, I have yet to
fully create the bots, I’m working on the first bot, and working on making
it so it follows robots.txt and robots META tags, and working on the back
end the options of what to use for the back end, elastic search, DB system
like MySQL or alike, or even a custom engine design, I have a lot of work
ahead of me and I’m working hard on it, I’m also working hard on the page
ranking algorithm aswell which will be hard to do. If people are wanting to
help with this I would be very happy to accept help on this project in any
form.

Sent from Mail for Windows 10

_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: Introduction of a sitemap style system for users and land

Grid Master
The other thing to consider, is with all these new privacy laws, and even
some existing ones, scraping and republishing data from a website you don't
own, could be a grey area as far as the letter of the law is concerned.
 Having the data pushed to you is going to mean that the person is giving
their consent willingly.



On Wed, Jul 4, 2018 at 9:28 AM, Cinder Roxley <[hidden email]>
wrote:

> Have you seen the functionality in Core that drives MetaverseInk’s web
> search already?
> http://www.metaverseink.com/blog/news/metaverse-ink-search-for-opensims/ I
> would suggest leveraging that for land instead. Crawling seems like a waste
> of resources when a grid could simply push the data to you in a format
> easily consumable. A module could be developed that pushes other updates to
> the search as well. (Does events work in Core? I don’t remember.) Failing
> that, a module that provides the information via a common api makes a lot
> more sense than scraping webpages.
>
> I wrote a search module a while back, I could open source it and you could
> build upon it with an external api that would provide consumers with
> relevant data in json or whatnot. Although, I still consider a push data
> model to the the more efficient design than poll and scrape.
>
> On July 4, 2018 at 7:56:44 AM, Alan Johnston (
> [hidden email]) wrote:
>
> I did not receive the reply I think I set my options wrong in the mailman
> but I did see the reply in the online archives, Haravikk The point of
> sitemaps is you submit then to a search engine for a search engine to have
> a set of URLs for them to look up and get the information from. Its a map
> to the site so to speak, this is good for such things as social sites, that
> wont have links on the main site that point to every possible page there
> is, as an example if you go to https://www.allvirtualsocial.com you will
> not see my post I posted on there a few weeks ago or the post about me
> creating android apps that are now available on the google play, but if you
> look at the sitemap https://www.allvirtualsocial.com/sitemap.xml you will
> see the link in there, although search engines dont use them for the main
> system to provide ranking, they use it to know where to crawl on the site,
> especially for pages that are not that obvious to “spiders”, so site maps
> are more common than people think, but are used primarily for search
> engines, this is why I have thought about this a lot. The concept would be
> to provide the search engine with the information it needs to provide
> search results to a set type, and would make be able to provide the nest
> answers it can give. I have a lot of work to do on the system including the
> bots, I’m thinking of making a 3 bot system 1 that does sites in the
> general sense, I.e. my social site, grids websites, news sites like Hyper
> Grid News, Blog sites related to virtual worlds and so forth, one bot to do
> site map collection for such sites, and another bot to do the land and
> users maps in grids, I’m going to install OS on my system some point today
> and check out some things on the system, and going to make a PHP sitemap
> Implementation of landmaps. And look at something when it comes to
> usermaps, though I would need to speak to someone within OS and Viewers to
> see if a thought is possible though I’m doubtful, this is something I am
> hopeing that can either be a module (and someone can help me create a
> opensource module of this) or even have it as a core integration, and have
> it with some config options so then people can choose to opt in or out of
> it.
> The search engine is a long way off from being finished, I have yet to
> fully create the bots, I’m working on the first bot, and working on making
> it so it follows robots.txt and robots META tags, and working on the back
> end the options of what to use for the back end, elastic search, DB system
> like MySQL or alike, or even a custom engine design, I have a lot of work
> ahead of me and I’m working hard on it, I’m also working hard on the page
> ranking algorithm aswell which will be hard to do. If people are wanting to
> help with this I would be very happy to accept help on this project in any
> form.
>
> Sent from Mail for Windows 10
>
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
> _______________________________________________
> Opensim-dev mailing list
> [hidden email]
> http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
>
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: Introduction of a sitemap style system for users and land

Michel Beauregard
In reply to this post by Cinder Roxley
The following   email present was  is already existing via Diva and other contibutors.

 search php  source code for access registered datasnapshot information:
<?php
$dbug=false;
$sval=mysql_real_escape_string($_GET["sname"]);
echo "searching for ", $sval," \n";
if ( strlen($sval) <3){
  echo  "need more then two letters to make search please \n";
  return;
}

if ( strlen($sval) >63){
  echo  "not more then 63  letters to make search please \n";
  return;
}
    
if (!preg_match("/^[a-zA-Z ]*$/",$sval)) {
  echo  "Only letters and white space allowed \n";
  return;
}

$sval="%".mysql_real_escape_string($_GET["sname"])."%";

$DB_HOST = "localhost";
$DB_USER = "opensim";
$DB_PASSWORD = "xxxxx";
$DB_NAME = "ossearch";

if ($dbug){
    echo $sval;
    echo "<br/>";
    echo $DB_HOST ;
    echo "<br/>";
    echo $DB_USER;
    echo "<br/>";
    echo $DB_PASSWORD;
    echo "<br/>";
    }

$conn = mysqli_connect ($DB_HOST, $DB_USER, $DB_PASSWORD, $DB_NAME);

if (mysqli_connect_errno())
  {
  echo "Failed to connect to MySQL: " . mysqli_connect_error();
  }

//$sql = "SELECT * FROM `objects` WHERE `name` LIKE \'%Tulip%\' LIMIT 0, 30 ";
//$sql = "SELECT * FROM `objects` WHERE `name` LIKE '%Tulip%'  LIMIT 0, 30 ";
//$sql = "SELECT * FROM `objects` INNER JOIN `regions` ON `objects.regionuuid` = `regions.regionuuid` LIMIT 0, 30 ";
//$sql = "SELECT `regionuuid` FROM `objects`";
//$sql = "SELECT `regionname`, `regionuuid`, `regionhandle`, `url`, `owner`, `owneruuid` FROM `regions` WHERE 1 LIMIT 0, 30 ";
$sql = "SELECT `location`, `name`, `description` , `regionname` FROM `objects` INNER JOIN `regions` ON objects.`regionuuid`=regions.`regionuuid` WHERE `name`  LIKE  '".  $sval . "' LIMIT 0, 300 ";

$result = $conn->query($sql);

if ($result->num_rows > 0) {
   echo "found ".$result->num_rows." item(s)  clic on the link below TWO times to teleport to object  \n";
    // output data of each row
    while($row = $result->fetch_assoc()) {
//        echo "Name: " . $row["name"]. " - Description: " . $row["description"]. " RegionName" . $row["regionname"]." Location" . $row["location"]. "<br>";
        echo "NAME: " . $row["name"]. " - DESC: " . $row["description"]. "  - LOC  https://gimisa/region/" . $row["regionname"]."/" . $row["location"];
    }
} else {
    echo "0 results";
}
$conn->close();
?>

As explain here http://metaverseink.com/docs/listing.html  To create your owne address search location using the above is as  simple as populating the opensim.ini file with  your site data. For exemple for my location  (3d.gimisa.ca:9000 ) the opensim.ini dataSnapshot  configuration is changed as follow with data_services pointing to metaverse search location in addition to my own internal location search facility :

[DataSnapshot]
    index_sims = true
    data_exposure = minimum

    gridname = "GiMiSa"
    ; 1 hour
    default_snapshot_period = 13600
    snapshot_cache_directory = "DataSnapshot"
    data_services="http://192.168.2.105/register.php;http://metaverseink.com/cgi-bin/register.py"

The data_service register.php code is as follows:
<?php
//////////////////////////////////////////////////////////////////////////////
// register.php                                                             //
// (C) 2008, Fly-man-                                                       //
// This file contains the registration of a simulator to the database       //
// and checks if the simulator is new in the database or a reconnected one  //
//                                                                          //
// If the simulator is old, check if the nextcheck date > registration      //
// When the date is older, make a request to the Parser to grab new data    //
//////////////////////////////////////////////////////////////////////////////

include("databaseinfo.php");
//establish connection to master db server
mysql_connect ($DB_HOST, $DB_USER, $DB_PASSWORD);
mysql_select_db ($DB_NAME);

$hostname = $_GET['host'];
$port = $_GET['port'];
$service = $_GET['service'];

if ($hostname != "" && $port != "" && $service == "online")
{
    // Check if there is already a database row for this host
    $checkhost = mysql_query("SELECT register FROM hostsregister WHERE " .
            "host = '" . mysql_real_escape_string($hostname) . "' AND " .
            "port = '" . mysql_real_escape_string($port) . "'");

    // Get the request time as a timestamp for later
    $timestamp = $_SERVER['REQUEST_TIME'];

    // if greater than 1, check the nextcheck date
    if (mysql_num_rows($checkhost) > 0)
    {
        $update = "UPDATE hostsregister SET " .
                "register = '" . mysql_real_escape_string($timestamp) . "', " .
                "nextcheck = '0', checked = '0', " .
                "failcounter = '0' " .  
                "WHERE host = '" . mysql_real_escape_string($hostname) . "' AND " .
                "port = '" . mysql_real_escape_string($port) . "'";

        $runupdate = mysql_query($update);
    }
    else
    {
        $register = "INSERT INTO hostsregister VALUES ".
                    "('" . mysql_real_escape_string($hostname) . "', " .
                    "'" . mysql_real_escape_string($port) . "', " .
                    "'" . mysql_real_escape_string($timestamp) . "', 0, 0, 0)";

        $runupdate = mysql_query($register);
    }
}
elseif ($hostname != "" && $port != "" && $service = "offline")
{
        $delete = "DELETE FROM hostsregister " .
                "WHERE host = '" . mysql_real_escape_string($hostname) . "' AND " .
                "port = '" . mysql_real_escape_string($port) . "'";

        $rundelete = mysql_query($delete);
}
?>

More detail about diva search can be found here: http://www.metaverseink.com/blog/news/metaverse-ink-search-for-opensims/ . Please note Metaverse search facility is broken since about begining of the year as it was before and  as it was never intended for full fledge support as opensim searching tool.

This article in hypergrid business give Maria's reflection on the subject in 2014  https://www.hypergridbusiness.com/2014/07/two-ideas-for-better-search-for-opensim/

(Happy 4th of July to our Americain friends .)

hope the above is  helpfull. GiMiSa
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: diva search tool

Michel Beauregard
In reply to this post by Grid Master
I forgot to mention the search option of metaversinkl.  That  2007 web page  is not that easy to find :
http://metaverseink.com/docs/tips.html

Interesting is the searcheable elements: product, place,sim, notecard please also note that the description field of an objject can be search.
Naturaly the php coding posted earlier does not offer that depth of searching but ones know that the information is available.
GiMiSa

_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: Introduction of a sitemap style system for users and land

Kevin Cozens
In reply to this post by Michel Beauregard
On 2018-07-04 04:21 PM, Michel Beauregard wrote:
> The following   email present was  is already existing via Diva and other contibutors.
>
>  search php  source code for access registered datasnapshot information:

That first part is typical of an addon search module (e.g. OpenSimSearch
module) that populates a database from information parsed from data created
by the DataSnapshot module. A third party could access the saved data using
an additional PHP script but that would still be grid specific.

> More detail about diva search can be found here: http://www.metaverseink.com/blog/news/metaverse-ink-search-for-opensims/ . Please note Metaverse search facility is broken since about begining of the year as it was before and  as it was never intended for full fledge support as opensim searching tool.

The MI search module from diva is more like what you want for a grid
agnostic search feature. The problem becomes one of deciding who will be the
host for all the aggregate data from the various grids.

--
Cheers!

Kevin.

http://www.ve3syb.ca/               | "Nerds make the shiny things that
https://www.patreon.com/KevinCozens | distract the mouth-breathers, and
                                     | that's why we're powerful"
Owner of Elecraft K2 #2172          |
#include <disclaimer/favourite>     |             --Chris Hardwick
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
Reply | Threaded
Open this post in threaded view
|

Re: Introduction of a sitemap style system for users and land

Michel Beauregard
Kevin metaversink is not grid specific.

I think correct me there if I am wrong please.

Datasnapshot is simulator driven.

If any sim  would be so kind as to  register there simulator  to me using the address provided in datasnapshot registration configuration of opensim.ini of my earlier post. They would be   pushing  to me  their datasnapshot information for their simulator so configured. My search result coverage would thefefor be extended to include their information irrespective of the grid they are part of .

If I dont make error in my understand of how this work in my setup,  this process is  simulator specific and not grid specific. So simulator owner ARE free to join whatever search facility ( none or many  )  they feel serve best their interest. 

( please dont try that I dont have the necessary resource  to support this sort of service )
 

GiMiSa

    Le jeudi 5 juillet 2018 11:41:15 HAE, Kevin Cozens <[hidden email]> a écrit :  
 
 On 2018-07-04 04:21 PM, Michel Beauregard wrote:
> The following   email present was  is already existing via Diva and other contibutors.
>
>  search php  source code for access registered datasnapshot information:

That first part is typical of an addon search module (e.g. OpenSimSearch
module) that populates a database from information parsed from data created
by the DataSnapshot module. A third party could access the saved data using
an additional PHP script but that would still be grid specific.

> More detail about diva search can be found here: http://www.metaverseink.com/blog/news/metaverse-ink-search-for-opensims/ . Please note Metaverse search facility is broken since about begining of the year as it was before and  as it was never intended for full fledge support as opensim searching tool.

The MI search module from diva is more like what you want for a grid
agnostic search feature. The problem becomes one of deciding who will be the
host for all the aggregate data from the various grids.

--
Cheers!

Kevin.

http://www.ve3syb.ca/               | "Nerds make the shiny things that
https://www.patreon.com/KevinCozens | distract the mouth-breathers, and
                                    | that's why we're powerful"
Owner of Elecraft K2 #2172          |
#include <disclaimer/favourite>    |            --Chris Hardwick
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev
 
_______________________________________________
Opensim-dev mailing list
[hidden email]
http://opensimulator.org/cgi-bin/mailman/listinfo/opensim-dev