Today I’m posting another technical hint for those who might want to extract huge dataset and have to cope with limited disk space issues. SQL Loader is a powerful Oracle tool for bulk loading, and so far the fastest on Oracle.
A major issue is surely the disk space that needs to be occupied while temporarly storing the loading data on local disks. This example will help you overcoming this issue by dumping the data direct on zip files and allowing SQL Loader to read directly from zipped files.

1. Prepare SQL select statement: this is the content inside the select.sql script

set pagesize 0
set head off
set feed off
set line 200

select
col1      || '|' ||
col2      || '|' ||
coln
from
SCHEMA.TABLENAME
;
exit;

2. Extract the dataset (for UNIX systems)

user@machine> mkfifo to-zip
user@machine> gzip -9 -c < to-zip > dataset.dat.gz
user@machine> sqlplus -s username/password@service @select.sql > to-zip

3. Create the SQLLDR control file (control.ctl file)

LOAD DATA
APPEND
INTO TABLE SCHEMA.TABLENAME
FIELDS TERMINATED BY '|'
TRAILING NULLCOLS
(
COL1
,COL2
,COLN
)

4. Load the dataset

user@machine> mkfifo unzip
user@machine> zcat dataset.dat.gz > unzip
user@machine> sqlldr userid=$username/$password@$service control=control.ctl log=results.log bad=results.bad discard=results.dsc data=unzip direct=y errors=0

And that’s it! please remember that a good behavior will be always to truncate the table first (if you need to replace the data) and force the rebuild of the table related indexes at the end of the process.

Share/Bookmark

In the last year I’ve spent six months digging into a PostgreSQL project. PostgreSQL is one of the most interesting and fast developed database management systems. However it is still a little bit behind the major commercial competitors, it is without any doubt, the most advanced OpenSource and Free DBMS. On the great features of PostgreSQL is the documentation of its architecture, that is maintained updated constantly by its vivid and active community. Although this community provides one of the best (the fastest that I’ve every experienced as a single user) developer on-line support system, at that time (and still now!) I had serious difficulties to find out information about on of the most used and useful coding extensions: custom types. Some of you may had to work with PostgreSQL extensions like PostGIS (the simplest and most effective spatial extension that integrates spatial information into PostgreSQL) and wanted to extend or to provide similar functionalities so easy to use and comfortable. I was mentioning the user-friendly constructors for spatial objects. Custom Types for defining spatial objects are defined in the SQL language like that

POINT('37','-97');

(for more information, take a look to this link)

This type of syntax is only possible (as far as I know) if the type is defined using a C function, and for instance compiled into a .so shared object library provided to PostgreSQL. C PostgreSQL custom types are defined by two functions that translate the low level description of the type into SQL usable constructors. Here it follows the code  to define constructors for a Currency type, and attached into this file, there is the source code for other operations like equals, greater and exchange rate calculation (the source of this piece of code is PostgreSQL C Reference Documentation).

/*
**  Name: fcur_in()
**
**        Converts an fcur value from external form
**	  to internal form.
*/

PG_FUNCTION_INFO_V1(fcur_in);

Datum fcur_in(PG_FUNCTION_ARGS)
{
    char  * src     = PG_GETARG_CSTRING(0);
    char  * workStr = (char *)palloc( strlen( src ));
    char  * units   = NULL;
    char  * name    = NULL;
    char  * xrate   = NULL;
    fcur  * result  = NULL;
    char  * endPtr  = NULL;

    strcpy( workStr, src );

    units = strtok( workStr, "(" );
    xrate = strtok( NULL, "/)" );
    name  = strtok( NULL, ")" );

    result = (fcur *)palloc( sizeof( fcur ));

    memset( result, 0x00, sizeof( fcur ));

    result->fcur_units = strtod( units, &endPtr );

    if( xrate )
    {
	result->fcur_xrate = strtod( xrate, &endPtr );
    }
    else
    {
	result->fcur_xrate = 1.0;
    }

    if( name )
    {
	strncpy( result->fcur_name,
		 name,
		 sizeof( result->fcur_name ));
    }
    else
    {
	strncpy( result->fcur_name,
		 unknownCurrencyName,
		 sizeof( result->fcur_name ));
    }

    PG_RETURN_POINTER( result );
}

/*
**  Name: fcur_out()
**
**        Converts an fcur value from internal form
**	  to external form.
*/

PG_FUNCTION_INFO_V1(fcur_out);

Datum fcur_out(PG_FUNCTION_ARGS)
{
    fcur  * src  = (fcur *)PG_GETARG_POINTER( 0 );
    char  * result;
    char    work[16+1+sizeof(src->fcur_name)+16];

    sprintf( work, "%g(%g/%s)",
	     src->fcur_units,
	     src->fcur_xrate,
	     src->fcur_name );

    result = (char *)palloc( strlen( work ) + 1 );

    strcpy( result, work );

    PG_RETURN_CSTRING( result );

}

Share/Bookmark