DeepLearning4j可以通過使用Apache Spark或者Hadoop來實現分布式訓練。下面是使用Apache Spark來實現分布式訓練的步驟:
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-core</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-ui_2.10</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.deeplearning4j</groupId>
<artifactId>deeplearning4j-scaleout</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-native</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.nd4j</groupId>
<artifactId>nd4j-cuda-9.2-platform</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-api</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-local</artifactId>
<version>1.0.0-beta3</version>
</dependency>
<dependency>
<groupId>org.datavec</groupId>
<artifactId>datavec-spark_2.10</artifactId>
<version>1.0.0-beta3</version>
</dependency>
SparkConf conf = new SparkConf();
conf.setAppName("DL4J Spark");
JavaSparkContext sc = new JavaSparkContext(conf);
JavaRDD<String> data = sc.textFile("hdfs://path/to/data.txt");
JavaRDD<DataSet> dataSet = data.map(new StringToDataSet());
MultiLayerConfiguration conf = new NeuralNetConfiguration.Builder()
.seed(12345)
.weightInit(WeightInit.XAVIER)
.updater(new Adam(0.01))
.list()
.layer(0, new DenseLayer.Builder().nIn(784).nOut(250)
.activation(Activation.RELU)
.build())
.layer(1, new OutputLayer.Builder(LossFunctions.LossFunction.NEGATIVELOGLIKELIHOOD)
.activation(Activation.SOFTMAX)
.nIn(250).nOut(10).build())
.build();
ComputationGraph model = new ComputationGraph(conf);
model.init();
SparkComputationGraph sparkNet = new SparkComputationGraph(sc, model);
sparkNet.fit(dataSet);
通過以上步驟,就可以使用DeepLearning4j和Apache Spark實現分布式訓練。同樣的,如果要使用Hadoop來實現分布式訓練,可以使用datavec-hadoop
依賴來讀取HDFS中的數據集,并使用SparkComputationGraph
對象進行訓練。