Today I’m posting another technical hint for those who might want to extract huge dataset and have to cope with limited disk space issues. SQL Loader is a powerful Oracle tool for bulk loading, and so far the fastest on Oracle.
A major issue is surely the disk space that needs to be occupied while temporarly storing the loading data on local disks. This example will help you overcoming this issue by dumping the data direct on zip files and allowing SQL Loader to read directly from zipped files.

1. Prepare SQL select statement: this is the content inside the select.sql script

set pagesize 0
set head off
set feed off
set line 200

select
col1      || '|' ||
col2      || '|' ||
coln
from
SCHEMA.TABLENAME
;
exit;

2. Extract the dataset (for UNIX systems)

user@machine> mkfifo to-zip
user@machine> gzip -9 -c < to-zip > dataset.dat.gz
user@machine> sqlplus -s username/password@service @select.sql > to-zip

3. Create the SQLLDR control file (control.ctl file)

LOAD DATA
APPEND
INTO TABLE SCHEMA.TABLENAME
FIELDS TERMINATED BY '|'
TRAILING NULLCOLS
(
COL1
,COL2
,COLN
)

4. Load the dataset

user@machine> mkfifo unzip
user@machine> zcat dataset.dat.gz > unzip
user@machine> sqlldr userid=$username/$password@$service control=control.ctl log=results.log bad=results.bad discard=results.dsc data=unzip direct=y errors=0

And that’s it! please remember that a good behavior will be always to truncate the table first (if you need to replace the data) and force the rebuild of the table related indexes at the end of the process.

Share/Bookmark

As soon as I’ve started digging into OLAP Architectures I realized how complicated things might become whenever looking for extra features. Most of the commercial OLAP solutions are providing “classic” OLAP features for data warehousing. It means that only simple numeric, strings and dates would be supported. Spatial Data Warehouses  architectures instead, are still in their infancy and there exists no commercial product that is actually providing such SOLAP functionalities. There exists several academic proposals instead which can provide a good overview on the features that can be supported by SOLAP Servers.

One of these acedemic prototypes is Geo Mondrian.

GeoMondrian is a spatially-enabled version of Pentaho Analysis Services (Mondrian). It has been released under the EPL. GeoMondrian is the first implementation of a true SOLAP server. It provides a consistent integration of spatial objects into the OLAP data cube structure, instead of fetching them from a separate spatial database, web service or GIS file. To make a simple analogy, GeoMondrian brings to the Mondrian OLAP server what PostGIS brings to the PostgreSQL database management system.

Unfortunately the documentation and getting-started guides are missing right now… That’s why I’ve decided to publish a couple of advices to whom may decide to use GeoMondrian for  developing SOLAP applications. Here attached you can find an archive that contains all that you need to setup a sample SOLAP server with geographic data source. It is designed to be working on PostgreSQL+Postigs since it is the only spatial database to be supported right now.

What you have to do in order to be able to use this unofficial build release of GeoMondrian is to load the .SQL script into your postgresql+postgis database and than run the Test application (be sure that the .XML schema definition is in the same parent directory, to have set up properly your JDBC connection and have included the libraries that are present in the “lib” folder).

I hope that it will help!

Share/Bookmark

GIS People often have to fight agains data conversions. The constellation of Desktop GIS suites make our life harder and harder while approaching to ETL processes. A relevant issue in ETL processes is the so called “data formats discovery”. Finding information about which software is supporting which spatial data type can turn a simple routing task into a “mission impossible” waist of time.

To overcome this inconvenient we can refer to the support guide of a famous ETL software: FME. FME is a complete spatial ETL solution that enables GIS Professionals to quickly translate, transform, integrate and distribute spatial data. In our case it simplifies our life by means of the “Supported Formats” guide.

Visiting this link [WWW] you may find a list of supported types from this software (quite a huge list!), however by clicking on each single data types you will be able to access to PDF version data sheets of each data source, complete of description, supported types, versions, and lot more.

Share/Bookmark

In the last year I’ve spent six months digging into a PostgreSQL project. PostgreSQL is one of the most interesting and fast developed database management systems. However it is still a little bit behind the major commercial competitors, it is without any doubt, the most advanced OpenSource and Free DBMS. On the great features of PostgreSQL is the documentation of its architecture, that is maintained updated constantly by its vivid and active community. Although this community provides one of the best (the fastest that I’ve every experienced as a single user) developer on-line support system, at that time (and still now!) I had serious difficulties to find out information about on of the most used and useful coding extensions: custom types. Some of you may had to work with PostgreSQL extensions like PostGIS (the simplest and most effective spatial extension that integrates spatial information into PostgreSQL) and wanted to extend or to provide similar functionalities so easy to use and comfortable. I was mentioning the user-friendly constructors for spatial objects. Custom Types for defining spatial objects are defined in the SQL language like that

POINT('37','-97');

(for more information, take a look to this link)

This type of syntax is only possible (as far as I know) if the type is defined using a C function, and for instance compiled into a .so shared object library provided to PostgreSQL. C PostgreSQL custom types are defined by two functions that translate the low level description of the type into SQL usable constructors. Here it follows the code  to define constructors for a Currency type, and attached into this file, there is the source code for other operations like equals, greater and exchange rate calculation (the source of this piece of code is PostgreSQL C Reference Documentation).

/*
**  Name: fcur_in()
**
**        Converts an fcur value from external form
**	  to internal form.
*/

PG_FUNCTION_INFO_V1(fcur_in);

Datum fcur_in(PG_FUNCTION_ARGS)
{
    char  * src     = PG_GETARG_CSTRING(0);
    char  * workStr = (char *)palloc( strlen( src ));
    char  * units   = NULL;
    char  * name    = NULL;
    char  * xrate   = NULL;
    fcur  * result  = NULL;
    char  * endPtr  = NULL;

    strcpy( workStr, src );

    units = strtok( workStr, "(" );
    xrate = strtok( NULL, "/)" );
    name  = strtok( NULL, ")" );

    result = (fcur *)palloc( sizeof( fcur ));

    memset( result, 0x00, sizeof( fcur ));

    result->fcur_units = strtod( units, &endPtr );

    if( xrate )
    {
	result->fcur_xrate = strtod( xrate, &endPtr );
    }
    else
    {
	result->fcur_xrate = 1.0;
    }

    if( name )
    {
	strncpy( result->fcur_name,
		 name,
		 sizeof( result->fcur_name ));
    }
    else
    {
	strncpy( result->fcur_name,
		 unknownCurrencyName,
		 sizeof( result->fcur_name ));
    }

    PG_RETURN_POINTER( result );
}

/*
**  Name: fcur_out()
**
**        Converts an fcur value from internal form
**	  to external form.
*/

PG_FUNCTION_INFO_V1(fcur_out);

Datum fcur_out(PG_FUNCTION_ARGS)
{
    fcur  * src  = (fcur *)PG_GETARG_POINTER( 0 );
    char  * result;
    char    work[16+1+sizeof(src->fcur_name)+16];

    sprintf( work, "%g(%g/%s)",
	     src->fcur_units,
	     src->fcur_xrate,
	     src->fcur_name );

    result = (char *)palloc( strlen( work ) + 1 );

    strcpy( result, work );

    PG_RETURN_CSTRING( result );

}

Share/Bookmark