There's a feature that's disabled by default whic can speed this up considerably - batchedStatements.
The documentation for this isn't the easiest to find, but what you need is
<target name="load.data">
<dbunit driver="${your.driver}" url="${your.url}" userid="${your.user}" password="${your.password}" schema="${your.user}">
<dbconfig>
<property name="batchedStatements" value="true"/>
<property name="batchSize" value="500"/>
</dbconfig>
<operation type="INSERT" transaction="false" src="${basedir}/yourdata.xml">
</operation>
</dbunit>
</target>
For us turning on batchedStatements reduced the time taken for a typical import from around 90 seconds to around 5 seconds without any other changes. Yes, those numbers is correct.
Note - the documentation does say that it's turned off by default as it's not supported by some jdbc drivers, but we had no problems with the oracle jdbc thin driver.