Share/Bookmark

If the earth stood still, the oceans would gradually migrate toward the poles and cause land in the equatorial region to emerge. This would eventually result in a huge equatorial megacontinent and two large polar oceans. The line that delineates the areas that hydrologically contribute to one or the other ocean would follow the equator if the earth was a perfect ellipsoid. However, due to the significant relief of both the continents and the ocean floor, the hypothetical global divide between the areas that hydrologically contribute to one or another ocean deviates from the equator significantly. Analogous to the well-known U.S. Continental Divide, this would be the border separating two giant hemispherical watersheds of the new circumpolar oceans. Interestingly, the highest point on this global divide would not be the highest altitude on the entire globe. The highest elevation of the global divide in the Colombian Andes would be about 12,280 meters, whereas the altitudes of the famous equatorial volcanoes of Chimborazo (Ecuador) and Kilimanjaro (Tanzania) would be 13,615 and 12,786 meters, respectively. Both volcanoes happen not to be located on the global divide line. The lowest point on the new global dividing line, with an elevation of 2,760 meters, would be situated southwest of Kiribati Island in the western Pacific.

(src Witold Fraczek, Esri)

Share/Bookmark

The report also notes that the Google Android platform continues to show rapid growth and over the past six months has edged past the Apple iPhone platform with new subscribers. But even with the latest spike, the iPhone still remains the smartphone that is most coveted by consumers.

(src www.nytimes.com)

Share/Bookmark

With the development of cartography, the history of landscapes has taken more and more importance to delineate the development of social settlements. As cited from [1] the illustration, let us take the famous depiction of Napoleon’s march on Moscow published by Charles Joseph Minard in 1861 ( a detailed description can be found in Tufte ). The author engages the readers in the exploration of the fate of Napoleon’s army in the Russian campaign of 1812-1813. Beginning at the Polish-Russian border, the thick band shows the size of the army at each position. The path of Napoleon’s retreat from Moscow in the cold winter is depicted by the dark lower band, which is tied to temperature and time scales. Tufte identified six separate variables that were shown within Minard’s drawing. First, the line width continuously marked the size of the army. Second and third, the line itself showed the latitude and longitude of the army as it moved. Fourth, the lines themselves showed the direction that the army was travelling, both in advance and retreat. Fifth, the location of the army with respect to certain dates was marked. Finally, the temperature along the path of retreat was displayed. It can also be noted that, despite of the schematic character of the drawing with its rudimentary cartography, Minard depicted some features of the underlying territory (specifically, rivers and towns) he deemed essential for the understanding of the story.

(src Adrienko et al.)

Share/Bookmark

Today I’m posting another technical hint for those who might want to extract huge dataset and have to cope with limited disk space issues. SQL Loader is a powerful Oracle tool for bulk loading, and so far the fastest on Oracle.
A major issue is surely the disk space that needs to be occupied while temporarly storing the loading data on local disks. This example will help you overcoming this issue by dumping the data direct on zip files and allowing SQL Loader to read directly from zipped files.

1. Prepare SQL select statement: this is the content inside the select.sql script

set pagesize 0
set head off
set feed off
set line 200

select
col1      || '|' ||
col2      || '|' ||
coln
from
SCHEMA.TABLENAME
;
exit;

2. Extract the dataset (for UNIX systems)

user@machine> mkfifo to-zip
user@machine> gzip -9 -c < to-zip > dataset.dat.gz
user@machine> sqlplus -s username/password@service @select.sql > to-zip

3. Create the SQLLDR control file (control.ctl file)

LOAD DATA
APPEND
INTO TABLE SCHEMA.TABLENAME
FIELDS TERMINATED BY '|'
TRAILING NULLCOLS
(
COL1
,COL2
,COLN
)

4. Load the dataset

user@machine> mkfifo unzip
user@machine> zcat dataset.dat.gz > unzip
user@machine> sqlldr userid=$username/$password@$service control=control.ctl log=results.log bad=results.bad discard=results.dsc data=unzip direct=y errors=0

And that’s it! please remember that a good behavior will be always to truncate the table first (if you need to replace the data) and force the rebuild of the table related indexes at the end of the process.

Share/Bookmark

I’m posting today some source code about the famous TomTom One .OV2 file formats. These files are the one used by TomTom to store Point of Interest information within their navigation systems. However such format is proprietary, the data specs can be found in the TomTom website (see this PDF at section 2.4). I wrote down a simple procedure that gathers record by record only the simple POI records. Actually the OV2 format is a little bit more complicated than a simple list of POIs and it can be subjected to changes and version incompatibilities. I tested it with POIs taken from my TomTom ONE XL and ti worked great, however if the OV2 files contains proprietary or special record types (not specified in TomTom data sheets) these procedures will simply stop since it is not possible for this format to distinguish between a corrupted file or non standard encoding.

I hope you’ll find it interesting.

Java version:

package readers;

import java.io.FileInputStream;
import java.io.IOException;

public class OV2RecordReader {

	public static String[] readOV2Record(FileInputStream inputStream){
		String[] record = null;
		int b = -1;
		try{
			if ((b = inputStream.read())> -1) {
				// if it is a simple POI record
				if (b == 2) {
					record = new String[3];
		    	    long total = readLong(inputStream);

		    	    double longitude = (double) readLong(inputStream) / 100000.0;
		    	    double latitude = (double) readLong(inputStream) / 100000.0;

		    	    byte[] r = new byte[(int) total - 13];
		    	    inputStream.read(r);

		    	    record[0] = new String(r);
		    	    record[0] = record[0].substring(0,record[0].length()-1);
		    	    record[1] = Double.toString(latitude);
		    	    record[2] = Double.toString(longitude);
		    	}
				//if it is a deleted record
				else if(b == 0){
					byte[] r = new byte[9];
		    	    inputStream.read(r);
				}
				//if it is a skipper record
				else if(b == 1){
					byte[] r = new byte[20];
		    	    inputStream.read(r);
				}
				else{
					throw new IOException("wrong record type");
				}
			}
			else{
				return null;
			}
		}
		catch(IOException e){
			e.printStackTrace();
		}
		return record;
	}

	private static long readLong(FileInputStream is){
		long res = 0;
		try{
			res = is.read();
			res += is.read() <<8;
			res += is.read() <<16;
			res += is.read() <<24;
		}
		catch(IOException e){
			e.printStackTrace();
		}
		return res;
	}
}

C# Version

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;

namespace OV2Reader
{
 class Program
 {
 static void Main(string[] args)
 {
 string filename = @"C:\test.ov2";
 FileStream infile = new FileStream(filename, FileMode.Open, FileAccess.Read);

 while (infile.Length != infile.Position)
 {
 string[] data = readOV2Record(infile);
 if (data != null)
 {
 for (int i = 0; i < data.Length; i++)
 {
 Console.WriteLine(data[i]);
 }
 }
 }

 Console.WriteLine("exiting....");
 Console.ReadKey();
 }

 static string[] readOV2Record(FileStream inputStream)
 {
 String[] record = null;
 int b = -1;

 if ((b = inputStream.ReadByte()) > -1)
 {
 // if it is a simple POI record
 if (b == 2)
 {

 long total = readLong(inputStream);

 double longitude = (double)readLong(inputStream) / 100000.0;
 double latitude = (double)readLong(inputStream) / 100000.0;

 byte[] r = new byte[(int)total - 13];
 inputStream.Read(r, 0, (int)total - 13);

 record = new String[3];
 string s = System.Text.ASCIIEncoding.ASCII.GetString(r);
 record[0] = s;
 record[0] = record[0].Substring(0, record[0].Length - 1);
 record[1] = latitude.ToString();
 record[2] = longitude.ToString();
 }
 //if it is a deleted record
 else if (b == 0)
 {
 byte[] r = new byte[9];
 inputStream.Read(r, 0, 9);
 }
 //if it is a skipper record
 else if (b == 1)
 {
 byte[] r = new byte[20];
 inputStream.Read(r, 0, 20);
 }
 else
 {
 //
 return null;
 }
 }
 else
 {
 return null;
 }

 return record;
 }

 private static long readLong(FileStream fis)
 {
 long res = 0;

 res = fis.ReadByte();
 res += fis.ReadByte() << 8;
 res += fis.ReadByte() << 16;
 res += fis.ReadByte() << 24;
 return res;
 }
 }
}

Share/Bookmark

As soon as I’ve started digging into OLAP Architectures I realized how complicated things might become whenever looking for extra features. Most of the commercial OLAP solutions are providing “classic” OLAP features for data warehousing. It means that only simple numeric, strings and dates would be supported. Spatial Data Warehouses  architectures instead, are still in their infancy and there exists no commercial product that is actually providing such SOLAP functionalities. There exists several academic proposals instead which can provide a good overview on the features that can be supported by SOLAP Servers.

One of these acedemic prototypes is Geo Mondrian.

GeoMondrian is a spatially-enabled version of Pentaho Analysis Services (Mondrian). It has been released under the EPL. GeoMondrian is the first implementation of a true SOLAP server. It provides a consistent integration of spatial objects into the OLAP data cube structure, instead of fetching them from a separate spatial database, web service or GIS file. To make a simple analogy, GeoMondrian brings to the Mondrian OLAP server what PostGIS brings to the PostgreSQL database management system.

Unfortunately the documentation and getting-started guides are missing right now… That’s why I’ve decided to publish a couple of advices to whom may decide to use GeoMondrian for  developing SOLAP applications. Here attached you can find an archive that contains all that you need to setup a sample SOLAP server with geographic data source. It is designed to be working on PostgreSQL+Postigs since it is the only spatial database to be supported right now.

What you have to do in order to be able to use this unofficial build release of GeoMondrian is to load the .SQL script into your postgresql+postgis database and than run the Test application (be sure that the .XML schema definition is in the same parent directory, to have set up properly your JDBC connection and have included the libraries that are present in the “lib” folder).

I hope that it will help!

Share/Bookmark

GIS People often have to fight agains data conversions. The constellation of Desktop GIS suites make our life harder and harder while approaching to ETL processes. A relevant issue in ETL processes is the so called “data formats discovery”. Finding information about which software is supporting which spatial data type can turn a simple routing task into a “mission impossible” waist of time.

To overcome this inconvenient we can refer to the support guide of a famous ETL software: FME. FME is a complete spatial ETL solution that enables GIS Professionals to quickly translate, transform, integrate and distribute spatial data. In our case it simplifies our life by means of the “Supported Formats” guide.

Visiting this link [WWW] you may find a list of supported types from this software (quite a huge list!), however by clicking on each single data types you will be able to access to PDF version data sheets of each data source, complete of description, supported types, versions, and lot more.

Share/Bookmark

Every time you start developing a new piece of software that needs external libraries to be used and integrated, the licensing dilemma strikes again. If you are a developer of a well standardized software company you can have stronger restrictions on the licenses to be used (even a list of usable licenses), however when you have to start your software from scratch, or start a software “business idea”, it is important to know something more on what you can take and what you can reuse without breaking somebody’s heart. I would like to dedicate this “in progress” post to collect in one page a brief description of the major families or licenses that can be used while developing free software but with possible commercial purposes! I hope it will help and I warmly advise anybody that would suggest any further detail to post a comment and I will be glad to integrate my content with that!

Apache License 2.0

• Free Software
• OSI Compatible
• Linking from code with a different license
• no Copyleft
• GPL v3 Compatible

software can be redistributed in any other license, but should be noticed that Apache software is included
Two files that must be put at the top directory of redistributed software packages:

* LICENSE – a copy of the license itself.
* NOTICE – A “notice” text document listing the names of licensed libraries used, together with their developers.

In every licensed file, any original copyright or patent notices in redistributed code must be preserved, and in every licensed file changed a notification must be added stating that changes have been made to that file.

Gnu LGPL License

• Free Software
• OSI Compatible
• Linking from code with a different license
• Copyleft
• GPL Compatible

an author may, through a copyleft licensing scheme, give every person who receives a copy of a work permission to reproduce, adapt or distribute the work as long as any resulting copies or adaptations are also bound by the same copyleft licensing scheme.

primarly used for software libraries; has copyleft restrictions. it is something in between GPL and BSD/APACHE like licenses.

BSD Licenses

• Free Software
• OSI Compatible
• Linking from code with a different license
• no Copyleft
• GPL Compatible

Eclipse Public License

• Free Software
• OSI Compatible
• Linking from code with a different license
• Copyleft
• no GPL  Compatible

is designed to be business friendly. allows developers to create their own licenses on the portion of the codes that are added.

Academic Free License

• Free Software
• OSI Compatible
• Linking from code with a different license
• no Copyleft
• GPL  Compatible unknown

considered redundant specifically to the Apache License 2.0

MIT License

considered a duplicate of BSD-like licenses

Share/Bookmark

In the last year I’ve spent six months digging into a PostgreSQL project. PostgreSQL is one of the most interesting and fast developed database management systems. However it is still a little bit behind the major commercial competitors, it is without any doubt, the most advanced OpenSource and Free DBMS. On the great features of PostgreSQL is the documentation of its architecture, that is maintained updated constantly by its vivid and active community. Although this community provides one of the best (the fastest that I’ve every experienced as a single user) developer on-line support system, at that time (and still now!) I had serious difficulties to find out information about on of the most used and useful coding extensions: custom types. Some of you may had to work with PostgreSQL extensions like PostGIS (the simplest and most effective spatial extension that integrates spatial information into PostgreSQL) and wanted to extend or to provide similar functionalities so easy to use and comfortable. I was mentioning the user-friendly constructors for spatial objects. Custom Types for defining spatial objects are defined in the SQL language like that

POINT('37','-97');

(for more information, take a look to this link)

This type of syntax is only possible (as far as I know) if the type is defined using a C function, and for instance compiled into a .so shared object library provided to PostgreSQL. C PostgreSQL custom types are defined by two functions that translate the low level description of the type into SQL usable constructors. Here it follows the code  to define constructors for a Currency type, and attached into this file, there is the source code for other operations like equals, greater and exchange rate calculation (the source of this piece of code is PostgreSQL C Reference Documentation).

/*
**  Name: fcur_in()
**
**        Converts an fcur value from external form
**	  to internal form.
*/

PG_FUNCTION_INFO_V1(fcur_in);

Datum fcur_in(PG_FUNCTION_ARGS)
{
    char  * src     = PG_GETARG_CSTRING(0);
    char  * workStr = (char *)palloc( strlen( src ));
    char  * units   = NULL;
    char  * name    = NULL;
    char  * xrate   = NULL;
    fcur  * result  = NULL;
    char  * endPtr  = NULL;

    strcpy( workStr, src );

    units = strtok( workStr, "(" );
    xrate = strtok( NULL, "/)" );
    name  = strtok( NULL, ")" );

    result = (fcur *)palloc( sizeof( fcur ));

    memset( result, 0x00, sizeof( fcur ));

    result->fcur_units = strtod( units, &endPtr );

    if( xrate )
    {
	result->fcur_xrate = strtod( xrate, &endPtr );
    }
    else
    {
	result->fcur_xrate = 1.0;
    }

    if( name )
    {
	strncpy( result->fcur_name,
		 name,
		 sizeof( result->fcur_name ));
    }
    else
    {
	strncpy( result->fcur_name,
		 unknownCurrencyName,
		 sizeof( result->fcur_name ));
    }

    PG_RETURN_POINTER( result );
}

/*
**  Name: fcur_out()
**
**        Converts an fcur value from internal form
**	  to external form.
*/

PG_FUNCTION_INFO_V1(fcur_out);

Datum fcur_out(PG_FUNCTION_ARGS)
{
    fcur  * src  = (fcur *)PG_GETARG_POINTER( 0 );
    char  * result;
    char    work[16+1+sizeof(src->fcur_name)+16];

    sprintf( work, "%g(%g/%s)",
	     src->fcur_units,
	     src->fcur_xrate,
	     src->fcur_name );

    result = (char *)palloc( strlen( work ) + 1 );

    strcpy( result, work );

    PG_RETURN_CSTRING( result );

}

Share/Bookmark