Importing Data into Neo4j

Importing data into Neo4j has two kinds of conditions, either installed into a fresh new instance of neo4j server, or into an existing server.
Into an existing server requires more effort and it is should be the technically challenging issue.
In this post, we only focus on the former.

Maxdemarzi’s blog left us a detailed steps of importing data as a fresh start, with the tools provided by Michael Hunger.

It should work as smooth as you could expect.
I am just going to log what I found troublesome.

Following above tutorial, I run the data generation test in my intelliJ IDE, to get around with class path/lib etc.
I spent 2475 s ~40 mins for 1 M nodes (21M size) and 25.6M relations (881M size) on my 8Gb virtual machine.
I mean it is not bad for fresh start, but how about for existing database? I am quite concerned with that.

After moving the produced data as data/graph.db and restart the server, I got this error message “Failed to start within… Check the logs”.
Going into the logs, I read that

Caused by: org.neo4j.kernel.StoreLockException: Unable to obtain lock on store lock file: /opt/neo4j-community-1.9/data/graph.db/store_lock
at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:94)
at org.neo4j.kernel.StoreLockerLifecycleAdapter.start(StoreLockerLifecycleAdapter.java:40)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:491)
… 12 more
Caused by: java.io.FileNotFoundException: /opt/neo4j-community-1.9/data/graph.db/store_lock (Permission denied)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.(RandomAccessFile.java:233)
at org.neo4j.kernel.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:53)
at org.neo4j.kernel.StoreLocker.checkLock(StoreLocker.java:80)
… 14 more

I went to the git for searching the source code, and identified one line causing the error.

storeLockFileChannel = fileSystemAbstraction.open( storeLockFile, "rw" );

It sounded like neo4j was failed in opening a “stock_lock” file with rw model.

The next thing I checked was to discover that by moving the generated data using sudo, it was endorsed as root.
Neo4j is accessing the folder under neo4j user privilege. No wonder it hit “permission denied”.

Solution is simple, moving the generated data as neo4j user, by “sudo su neo4j”.
OK, restarting the server the problem was solved. I feel that I need to spend sometime to read the internals of neo4j.

More exceptions after batch-import

Problem:

org.neo4j.graphdb.NotFoundException: Target file[nioneo_logical.log.v0] already exists
at org.neo4j.kernel.impl.util.FileUtils.renameFile(FileUtils.java:192)
at org.neo4j.kernel.DefaultFileSystemAbstraction.renameFile(DefaultFileSystemAbstraction.java:77)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameLogFileToRightVersion(XaLogicalLog.java:683)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameIfExists(XaLogicalLog.java:181)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:150)

Solution:
This is the neo4j file for handling rolling back consistency or crashing safely.
Simply remove the file.

Problem:

Caused by: org.neo4j.graphdb.NotFoundException: Target file[lucene.log.v0] already exists
at org.neo4j.kernel.impl.util.FileUtils.renameFile(FileUtils.java:192)
at org.neo4j.kernel.DefaultFileSystemAbstraction.renameFile(DefaultFileSystemAbstraction.java:77)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameLogFileToRightVersion(XaLogicalLog.java:683)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.renameIfExists(XaLogicalLog.java:181)
at org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog.open(XaLogicalLog.java:150)
at org.neo4j.kernel.impl.transaction.xaframework.XaContainer.openLogicalLog(XaContainer.java:64)
at org.neo4j.index.impl.lucene.LuceneDataSource.(LuceneDataSource.java:232)
at org.neo4j.index.lucene.LuceneIndexProvider.load(LuceneIndexProvider.java:72)
at org.neo4j.kernel.InternalAbstractGraphDatabase$DefaultKernelExtensionLoader.loadIndexImplementations(InternalAbstractGraphDatabase.java:1171)
at org.neo4j.kernel.InternalAbstractGraphDatabase$DefaultKernelExtensionLoader.init(InternalAbstractGraphDatabase.java:1143)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.init(LifeSupport.java:382)
… 10 more

Solution:
Remove the conflicting file.

Advertisements
Tagged

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: