Add PlayTorch to Existing App

In this tutorial, you will learn how to add the PlayTorch core package to an existing React Native project.

If you have an existing React Native project and you want to add ML capabilities, you can add the react-native-pytorch-core package. This package includes all code needed to run ML inference, the Canvas, Camera, ImageUtil, and AudioUtil.

Installation

yarn add react-native-pytorch-core

caution

If you are on React Native 0.68+, you will have to use the nightly version of the core package. A fix has been added in commit fix RN0.68 can't build with react-native-pytorch-core, which is currently only available in the nightly version.

yarn add react-native-pytorch-core@nightly

On iOS you are done, but Android requires the following additional steps for the react-native-pytorch-core package to work.

Additional Assets for Metro

If the PyTorch Mobile models are part of the React Native bundle, the Metro configuration needs to be changed to resolve the ptl files.

note

This is only required if models are loaded from the bundle using require('./path/to/model.ptl'). It is not required if models are loaded from the mobile device's local file system or via a URL.

metro.config.js

// get defaults assetExts array
const defaultAssetExts = require('metro-config/src/defaults/defaults')
  .assetExts;

module.exports = {
  // ...

  resolver: {
    assetExts: [...defaultAssetExts, 'ptl'],
  },

  // ...
};

Additional steps on Android

For the react-native-pytorch-core React Native package to work on Android, it requires three changes to the gradle.properties and the two build.gradle files to increase JVM memory, add Sonatype repository, and packaging options with pick first rule.

Increase JVM Memory

Increase the memory for the JVM to avoid OutOfMemory exceptions during the packaging process.

./android/gradle.properties

org.gradle.jvmargs=-Xmx4g

Without the increased memory, the packaging process might fail with the following error:

* What went wrong:
Execution failed for task ':app:packageDebug'.
> A failure occurred while executing com.android.build.gradle.tasks.PackageAndroidArtifact$IncrementalSplitterRunnable
   > java.lang.OutOfMemoryError (no error message)

Sonatype Repository

The PyTorch Mobile for Android dependencies are in the Sonatype repository. Add the repository url to the allprojects > repositories.

./android/build.gradle

allprojects {
    repositories {
        // ...

        maven {
          url("https://oss.sonatype.org/content/repositories/snapshots")
        }

        // ...
    }
}

Update app build.gradle

Both React Native and PyTorch Mobile for Android use fbjni. For example, the versions for PlayTorch that are used for development are:

• React Native 0.64.3 uses fbjni 0.0.2
• PyTorch Mobile 1.12.2 uses fbjni 0.2.2.

So far, fbjni is forward compatible, which means it is ok to pick the latest version shipped with either of the two dependencies. At this point, it is fbjni 0.2.2. For Gradle to pick the right version, the android/app/build.gradle needs to have a few adjustments:

1. Add pickFirst rule to packagingOptions. This rule will pick the first shared object (dynamic) library. It will give higher priority to shared object libraries that are coming with direct app dependencies, which is why 2. is important.
2. Set up an extra directory for fbjni where the fbjni version 0.2.2 from the dependency added in 3. will be extracted. Also add the relevant task to the build.gradle file (see task extraJNILibs and tasks.whenTaskAdded after the dependencies definition)
3. Add fbjni 0.2.2 as direct app dependency.

See the build.gradle in the react-native-pytorch-core example app for a possible configuration.

info

The following error will show if pickFirst is not set:

* What went wrong:
Execution failed for task ':app:mergeDebugNativeLibs'.
> A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
   > More than one file was found with OS independent path 'lib/x86/libfbjni.so'

./android/app/build.gradle

android {
    // ...

    /**
     * Without the packaging options, it will result in the following build error:
     *
     * * What went wrong:
     * Execution failed for task ':app:mergeDebugNativeLibs'.
     * > A failure occurred while executing com.android.build.gradle.internal.tasks.Workers$ActionFacade
     *    > More than one file was found with OS independent path 'lib/x86/libfbjni.so'
     */
    packagingOptions {
        pickFirst '**/*.so'
    }
    sourceSets {
        main {
            jniLibs.srcDirs += ["$buildDir/extra-jniLibs/jni"]
        }
    }
    configurations {
        extraJNILibs
    }

    // ...
}

dependencies {
    // ...

    // Used to control the version of libfbjni.so packaged into the APK
    extraJNILibs("com.facebook.fbjni:fbjni:0.2.2")

    // ...
}

// ...

// Extract JNI shared libraries as project libraries. This assumes the target directory, $buildDir/extra-jniLibs, is added to the jniLibs.srcDirs configuration.
task extraJNILibs {
  doLast {
    configurations.extraJNILibs.files.each {
      def file = it.absoluteFile

      copy {
        from zipTree(file)
        into "$buildDir/extra-jniLibs" // temp location instead of "src/main/jniLibs"
        include "jni/**/*"
      }
    }
  }
}

tasks.whenTaskAdded { task ->
  if (task.name == 'mergeDebugJniLibFolders' || task.name == 'mergeReleaseJniLibFolders') {
    task.dependsOn(extraJNILibs)
  }
}

// ...

Configure JavaScript Interface for React Native app

Last, add the PyTorchCoreJSIModulePackage to the ReactNativeHost in the app's MainApplication.java.

info

JavaScript Interface (JSI)

A lightweight and VM-independent layer that helps in communication between JavaScript and native platforms easier and faster. It supports Webkit JSC, Custom JSC, and Hermes.

JSI Documentation

The jsi.h in the Hermes GitHub repository and has a really well-documented C++ header, which is a great place to learn more about JSI.

PlayTorch uses the JavaScript Interface (JSI) to expose PyTorch Mobile C++ functions

./android/app/src/main/java/<your-package-path>/MainApplication.java

import com.facebook.react.bridge.JSIModulePackage;
import org.pytorch.rn.core.jsi.PyTorchCoreJSIModulePackage;

// ...

      new ReactNativeHostWrapper(
          this,
          new ReactNativeHost(this) {
            // ...

            @Override
            protected JSIModulePackage getJSIModulePackage() {
              return new PyTorchCoreJSIModulePackage();
            }
          });

Great! You are done if you don't use other React Native packages that rely on JSI, otherwise continue!

Configure multiple React Native packages using JSI

If you use other React Native packages with JSI packages, you will have to create a JSIModulePackage that combines all of them. For example, if you use React Reanimated and PlayTorch Core create a CustomJSIModulePackage and return this in the ReactNativeHost implementation.

./android/app/src/main/java/<your-package-path>/CustomJSIModulePackage.java

package <your-package-path>;

import androidx.annotation.Keep;
import com.facebook.proguard.annotations.DoNotStrip;
import com.facebook.react.bridge.JSIModulePackage;
import com.facebook.react.bridge.JSIModuleSpec;
import com.facebook.react.bridge.JavaScriptContextHolder;
import com.facebook.react.bridge.ReactApplicationContext;
import com.facebook.soloader.SoLoader;
import com.swmansion.reanimated.ReanimatedJSIModulePackage;
import java.util.List;
import org.pytorch.rn.core.jsi.PyTorchCoreJSIModulePackage;

/**
 * This is a custom JSIModulePackage that combines ReanimatedJSIModulePackage and
 * PyTorchCoreJSIModulePackage. It is needed because the ReactNativeHostWrapper only supports
 * returning a single JSIModulePackage.
 */
@DoNotStrip
@Keep
public class CustomJSIModulePackage implements JSIModulePackage {

  @DoNotStrip
  @Keep
  @Override
  public List<JSIModuleSpec> getJSIModules(
      ReactApplicationContext reactApplicationContext, JavaScriptContextHolder jsContext) {
    ReanimatedJSIModulePackage reaJSIModulePackage = new ReanimatedJSIModulePackage();
    PyTorchCoreJSIModulePackage ptlJSIModulePackage = new PyTorchCoreJSIModulePackage();

    List<JSIModuleSpec> retList =
        reaJSIModulePackage.getJSIModules(reactApplicationContext, jsContext);
    ptlJSIModulePackage.getJSIModules(reactApplicationContext, jsContext);

    return retList;
  }
}

./android/app/src/main/java/<your-package-path>/MainApplication.java

// ...
import com.facebook.react.bridge.JSIModulePackage;

      new ReactNativeHostWrapper(
          this,
          new ReactNativeHost(this) {
            // ...

            @Override
            protected JSIModulePackage getJSIModulePackage() {
              // The CustomJSIModulePackage combines JSIModulePackage from different React Native
              // packages (e.g., Reanimated2 and PlayTorch).
              return new CustomJSIModulePackage();
            }
          });

That should be all necessary Gradle build configuration changes!

Give us feedback

Share what we can improve!