Talend, the open source company with the data integration software, is moving up-market with a new enterprise-grade Integration Suite MPx that won't be open sourced.

The massively parallelized widgetry will pit Talend against the ultra-secretive ETL house Ab Initio, which is notorious for not letting its software be seen by anyone other than NDA-sworn users, refusing interviews with Gartner and not returning phone calls.

The ploy appears to be part feminine mystique, part pure paranoia.

Ab Initio was started maybe 14 years ago by Sheryl Handler, a eccentric woman who studied interior design, has a master's in landscape architecture from Harvard, and wound up CEO of Thinking Machines, the seminal-but-doomed massively parallel supercomputer house and DARPA pet whose black Connection Machine made a cameo appearance in the movie Jurassic Park.

Anyway, Talend VP of marketing Yves de Montcheuil thinks that despite Ab Initio's cloak of secrecy, Integration Suite MPx's FileScale technology and massively parallelized processes will match its rival's reportedly chi-chi skills with huge amounts of data.

Talend reckons Integration Suite MPx can "shatter the limits typically associated with traditional data integration processes."

It's had mathematicians at the École Polytechnique, the great French engineering school, working on the MPx's underlying algorithms; MPx's development took 18 months.

The FileScale technology that MPx introduces to Talend's Integration Suite was inspired but not copied from Google's MapReduce architecture for highly distributed processing of data in a data set mode.

It lets integration processes sort, filter and merge data, do aggregation and arithmetic functions, and transform and ensure the compliance of data.

Talend says FileScale components are optimized for each supported platform for the highest possible performance. Those platforms include 32- and 64-bit Windows and Linux, Solaris and OpenSolaris, both Sparc and Intel, AIX and HP-UX.

The widgetry also has multiple levels of massive parallelization that can execute separate sub-processes in parallel, break down data sets into many parallel streams, and leverage parallel database loaders.

Talend is promising to process great big volumes of data very quickly to draw nearer to real-time meaning from it. It says data warehouses and analytic applications can be refreshed more often and include more granular data for more accurate analysis.

Talend ran a benchmark on a Sun Blade X6270 server fitted with two quad-core 2.2GHz Xeons and 24GB of RAM running Solaris 10 Update 6, and claims to have reached levels of scalability in sorting and calculation tests that have never been met before.

Of course there are no public benchmarks of Ab Initio, but Talend says its benchmark was done with industry-standard TPC-H data sets ranging from six million to 3.3 billion records and its in-memory sorting reached up to a million records a second with linear scalability and record-breaking sorting on the really big data sets of 200,000-400,000 records a second with an average of 220,000 records a second.

Talend's pricing depends on how many developers are using the stuff, starting at about $100,000. Ab Initio reportedly costs about $500,000 on average and may run into the millions. It is also supposed to eat up hardware resources. Both pieces of software heavily target financial houses and the telecom industry to streamline operations such as invoice generation.

Talend is backed by AGF Private Equity and Balderton Capital, an early investor in MySQL whose general partner Bernard Liautaud started and ran Business Objects until it was sold to SAP for $6.7 billion. Liautaud is now on Talend's board.