Base on this here is a tutorial to use PoseTracker in a Mobile App :
Start a new Expo project
npx create-expo-app my-app
cd my-app
yarn install
Install dependencies for the webview
npx expo install expo-camera react-native-webview
Start and run the project on your Phone or a Simulator
For this part we recommend to use your phone with Expo Go App as described in Expo Documentation. Because we will need to access a camera with a human to track.
yarn start
// Then open the app on Expo GO
Integrate PoseTracker
Define your App.js and set your API_KEY :
import { StyleSheet, Text, View, Platform } from 'react-native';
import {useState} from "react";
import WebView from "react-native-webview";
import { Camera } from 'expo-camera'
// Our API request your token provided on our dashboard on app.posetracker.com (It's free <3)
const POSETRACKER_API = "https://app.posetracker.com/pose_tracker/tracking"
export default function App() {
const [poseTrackerInfos, setCurrentPoseTrackerInfos] = useState()
const [repsCounter, setRepsCounter] = useState(0)
const [hasPermission, setHasPermission] = useState(false)
// Our API request the width and height wanted for display the webcam inside the webview
const width = 350
const height = 350
// Our API request the exercise you want to track and count
const exercise = "squat"
// Our API request the difficulty of the exercise (by default it's set to normal)
const difficulty = "easy"
// You can request API to display user skeleton or not (by default it's set to true)
const skeleton = true
const posetracker_url = `${POSETRACKER_API}?token=${API_KEY}&exercise=${exercise}&difficulty=${difficulty}&width=${width}&height=${height}`
// We need a bridge to transit data between the ReactNative app and our WebView
// The WebView will use this function define here to send info that we will use later
const jsBridge = `
(function() {
window.webViewCallback = function(info) {
window.ReactNativeWebView.postMessage(JSON.stringify(info));
}
})();
`
const handleCounter = (count) => {
setRepsCounter(count)
}
const handleInfos = (infos) => {
setCurrentPoseTrackerInfos(infos)
}
//This is the function pass to the WebView to listen info from the WebView
const webViewCallback = (info) => {
switch (info.type) {
case 'counter':
return handleCounter(info.current_count)
default:
return handleInfos(info)
}
}
// For Android it's needed to request camera authorization
useEffect(() => {
if (Platform.OS === 'android') {
;(async () => {
const { status } = await Camera.requestCameraPermissionsAsync()
setHasPermission(status === 'granted')
})()
}
}, []);
if (Platform.OS === 'android' && !hasPermission) {
return <HStack
alignItems='center'
justifyContent='center'
style={{ height: '100%', backgroundColor: 'black' }}>
<TopButtons onStop={onStop} marginLeft={20} marginRight={20}/>
<View width={width} height={height}>
<Text>{t('app.camera.need_access')}</Text>
</View>
</HStack>
}
return (
<View style={styles.container}>
<View style={{ flex: 1}}>
<WebView
javaScriptEnabled={true}
domStorageEnabled={true}
allowsInlineMediaPlayback={true}
mediaPlaybackRequiresUserAction={false}
style={{
width: width,
height: height,
zIndex: 1
}}
source={{ uri: posetracker_url }}
originWhitelist={['*']}
injectedJavaScript={jsBridge}
onMessage={(event) => {
const info = JSON.parse(event.nativeEvent.data)
webViewCallback(info)
}}
/>
</View>
<View style={{ flex: 1}}>
<Text>Status : {!poseTrackerInfos ? "loading AI..." : "AI Running"}</Text>
<Text>Info type : {!poseTrackerInfos ? "loading AI..." : poseTrackerInfos.type}</Text>
{ poseTrackerInfos?.ready === false ? (<>
<Text>Placement ready: false</Text>
<Text>Placement info : Move { poseTrackerInfos?.postureDirection } </Text>
</>) : (
<>
<Text>Placement ready: true</Text>
<Text>Placement info : You can start doing squats đī¸</Text>
</>
)}
{ repsCounter > 0 && (
<Text>Counter: {repsCounter}</Text>
)}
</View>
</View>
);
}
const styles = StyleSheet.create({
container: {
flex: 1,
flexDirection: 'column',
marginTop: 60,
},
});
That use posetracker_url who's build with PoseTracker params.
đ§ Important point : Data exchange between PoseTracker and your Application đ§
We use what you are sending throw injectedJavaScript to send back data to our App. So we create a bridge between PoseTracker API frontend and YOUR actual App with :
injectedJavaScript={jsBridge}
// This is a basic js bridge
// We need a bridge to transit data between the ReactNative app and our WebView
// The WebView will use this function define here to send info that we will use later
const jsBridge = `
(function() {
window.webViewCallback = function(info) {
window.ReactNativeWebView.postMessage(JSON.stringify(info));
}
})();
`
Then we use your bridge to send you some information, and here we handle it with:
onMessage={(event) => {
const info = JSON.parse(event.nativeEvent.data)
webViewCallback(info)
}}
const handleCounter = (count) => {
setRepsCounter(count)
}
const handleInfos = (infos) => {
setCurrentPoseTrackerInfos(infos)
}
//This is the function pass to the WebView to listen info from the WebView
const webViewCallback = (info) => {
switch (info.type) {
case 'counter':
return handleCounter(info.current_count)
default:
return handleInfos(info)
}
}