📱Expo React Native example

How to use PoseTracker in a react native application ?

Base on the schema on How does it work ? You need to call our URL https://app.posetracker.com/pose_tracker/tracking through a WebView.

Base on this here is a tutorial to use PoseTracker in a Mobile App :

Start a new Expo project

npx create-expo-app my-app
cd my-app
yarn install

Install dependencies for the webview

npx expo install expo-camera react-native-webview

Start and run the project on your Phone or a Simulator

For this part we recommend to use your phone with Expo Go App as described in Expo Documentation. Because we will need to access a camera with a human to track.

yarn start

// Then open the app on Expo GO

Integrate PoseTracker

Define your App.js and set your API_KEY :

import { StyleSheet, Text, View, Platform } from 'react-native';
import {useState} from "react";
import WebView from "react-native-webview";
import { Camera } from 'expo-camera'

// Our API request your token provided on our dashboard on app.posetracker.com (It's free <3)

const POSETRACKER_API = "https://app.posetracker.com/pose_tracker/tracking"

export default function App() {
  const [poseTrackerInfos, setCurrentPoseTrackerInfos] = useState()
  const [repsCounter, setRepsCounter] = useState(0)
  const [hasPermission, setHasPermission] = useState(false)

  // Our API request the width and height wanted for display the webcam inside the webview
  const width = 350
  const height = 350

  // Our API request the exercise you want to track and count
  const exercise = "squat"

  // Our API request the difficulty of the exercise (by default it's set to normal)
  const difficulty = "easy"

  // You can request API to display user skeleton or not (by default it's set to true)
  const skeleton = true

  const posetracker_url = `${POSETRACKER_API}?token=${API_KEY}&exercise=${exercise}&difficulty=${difficulty}&width=${width}&height=${height}`

  // We need a bridge to transit data between the ReactNative app and our WebView
  // The WebView will use this function define here to send info that we will use later
  const jsBridge = `
  (function() {
    window.webViewCallback = function(info) {
      window.ReactNativeWebView.postMessage(JSON.stringify(info));
    }
  })();
`

  const handleCounter = (count) => {
    setRepsCounter(count)
  }

  const handleInfos = (infos) => {
    setCurrentPoseTrackerInfos(infos)
  }

  //This is the function pass to the WebView to listen info from the WebView
  const webViewCallback = (info) => {
    switch (info.type) {
      case 'counter':
        return handleCounter(info.current_count)
      default:
        return handleInfos(info)
    }
  }

  // For Android it's needed to request camera authorization
  useEffect(() => {
    if (Platform.OS === 'android') {
      ;(async () => {
        const { status } = await Camera.requestCameraPermissionsAsync()
        setHasPermission(status === 'granted')
      })()
    }
  }, []);

  if (Platform.OS === 'android' && !hasPermission) {
    return <HStack
      alignItems='center'
      justifyContent='center'
      style={{ height: '100%', backgroundColor: 'black' }}>
      <TopButtons onStop={onStop} marginLeft={20} marginRight={20}/>
      <View width={width} height={height}>
        <Text>{t('app.camera.need_access')}</Text>
      </View>
    </HStack>
  }

  return (
    <View style={styles.container}>
      <View style={{ flex: 1}}>
        <WebView
        javaScriptEnabled={true}
        domStorageEnabled={true}
        allowsInlineMediaPlayback={true}
        mediaPlaybackRequiresUserAction={false}
        style={{
          width: width,
          height: height,
          zIndex: 1
        }}
        source={{ uri: posetracker_url }}
        originWhitelist={['*']}
        injectedJavaScript={jsBridge}
        onMessage={(event) => {
          const info = JSON.parse(event.nativeEvent.data)
          webViewCallback(info)
        }}
      />
      </View>
      <View style={{ flex: 1}}>
        <Text>Status : {!poseTrackerInfos ? "loading AI..." : "AI Running"}</Text>
        <Text>Info type : {!poseTrackerInfos ? "loading AI..." : poseTrackerInfos.type}</Text>
        { poseTrackerInfos?.ready === false ? (<>
          <Text>Placement ready: false</Text>
          <Text>Placement info : Move { poseTrackerInfos?.postureDirection } </Text>
        </>) : (
          <>
            <Text>Placement ready: true</Text>
            <Text>Placement info : You can start doing squats 🏋️</Text>
          </>
        )}
        { repsCounter > 0 && (
          <Text>Counter: {repsCounter}</Text>
        )}
      </View>
    </View>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    flexDirection: 'column',
    marginTop: 60,
  },
});

How does it works ?

• First we have the WebView :

<WebView
        javaScriptEnabled={true}
        domStorageEnabled={true}
        allowsInlineMediaPlayback={true}
        mediaPlaybackRequiresUserAction={false}
        style={{
          width: width,
          height: height,
          zIndex: 1
        }}
        source={{ uri: posetracker_url }}
        originWhitelist={['*']}
        injectedJavaScript={jsBridge}
        onMessage={(event) => {
          const info = JSON.parse(event.nativeEvent.data)
          webViewCallback(info)
        }}
      />

That use posetracker_url who's build with PoseTracker params.

🟧 Important point : Data exchange between PoseTracker and your Application 🟧

We use what you are sending throw injectedJavaScript to send back data to our App. So we create a bridge between PoseTracker API frontend and YOUR actual App with :

injectedJavaScript={jsBridge}

// This is a basic js bridge
// We need a bridge to transit data between the ReactNative app and our WebView
// The WebView will use this function define here to send info that we will use later
const jsBridge = `
 (function() {
   window.webViewCallback = function(info) {
     window.ReactNativeWebView.postMessage(JSON.stringify(info));
   }
 })();
`

Then we use your bridge to send you some information, and here we handle it with:

onMessage={(event) => {
  const info = JSON.parse(event.nativeEvent.data)
  webViewCallback(info)
}}
        
const handleCounter = (count) => {
  setRepsCounter(count)
}

const handleInfos = (infos) => {
  setCurrentPoseTrackerInfos(infos)
}

//This is the function pass to the WebView to listen info from the WebView
const webViewCallback = (info) => {
  switch (info.type) {
    case 'counter':
      return handleCounter(info.current_count)
    default:
      return handleInfos(info)
  }
}

You can find all the informations returned by PoseTracker here.

Result

Video of the tutorial App

You can find source files here

GitHub - Movelytics/ReactNativeWebcamDemo: A React Native application with expo and using PoseTracker API to count squatsGitHub