Data Pipeline 2: Data Transformation Toolkit for Java Released

Data Pipeline makes it easy to convert, process, and transform data in Java applications.

The new release includes...

• Open source (GPL) and commercial licenses
• New data readers, writers, and transformers
• Java 5 support

Features

• Readers: CSV, fixed-width, Excel, database, weblogs, custom
• Writers:  CSV, fixed-width, Excel, database, PDF, Word, XML, custom
• Sorting (single or multi-field with collation)
• External (disk-based) sorting for large data sets
• Filters (programmatic or using our run-time expression language)
• Data validation (programmatic or using our run-time expression language)
• Calculated fields (programmatic or using our run-time expression language)
• Field copying
• Field renaming
• Duplicate records
• Remove duplicate records (using selected fields or entire record)
• Field removal (black list/exclusion or white list/inclusion)
• Field selection and arrangement
• Field conversion and formatting (see BasicFieldTransformer for a better idea)
• Field aggregation (minimum, maximum, average, sum, & count)
• Lookups (from database, another data reader, or custom)
• Throttling and metering

A typical scenario might be to:

  1. read a CSV file
  2. remove duplicate records
  3.  add calculated field
  4. remove unused columns
  5. save to database

    public static void main(String[] args) throws Throwable {
        DataReader reader = new CSVReader(new File("credit-balance.csv"))
            .setFieldNamesInFirstRow(true);
        
        // Use only the "Rating" and "CreditLimit" fields in duplicate test
        reader = new RemoveDuplicatesReader(reader, new FieldList("Rating", "CreditLimit"));        
        
        // Add "AvailableCredit" field, remove "CreditLimit" and "Balance" fields
        reader = new TransformingReader(reader)
            .add(new SetCalculatedField("AvailableCredit", "parseDouble(CreditLimit) - parseDouble(Balance)"))
            .add(new ExcludeFields("CreditLimit", "Balance"));
        
        DataWriter writer = new  JdbcWriter(getJdbcConnection(), "dp_credit_balance")
            .setAutoCloseConnection(true);
        
        JobTemplate.DEFAULT.transfer(reader, writer);
    }

 
We look forward to hearing your feedback.

Downloads:
http://northconcepts.com/downloads/

Examples:
http://northconcepts.com/data-pipeline/examples/

Getting Started:
http://northconcepts.com/data-pipeline/getting-started/

Forums:
http://northconcepts.com/forums/


Dele Taylor
North Concepts Inc.